B RIDGING R ECOMMENDATION AND M ARKETING VIA R ECURRENT I NTENSITY M ODELS FOR E XPLORATION
A BSTRACT
This paper studies some unexplored connections between personalized recommendation and marketing systems.
Obviously, the two systems are different, in two main ways.
Firstly, personalized item-recommendation (ItemRec) is usercentric, whereas marketing recommends the best user-state segments (UserRec) on behalf of its item providers.
(We treat different temporal states of the same user as separate marketing opportunities.)
To overcome this difference, we realize a novel connection to Marked-Temporal Point Processes (MTPPs), where we view both problems as different projections from a uniﬁed temporal intensity model for all user-item pairs. In this way, we derive Recurrent Intensity Models (RIMs) as unifying extensions from recurrent ItemRec models, though the connection can be more general. The second difference is in the temporal domains where they operate.
While recommendation happens in real-time as each user appears, marketers often aim to reach a certain percentage of audience in the distribution of all userstates in a period of time. We formulate both considerations into a constrained optimization problem we call online match (OnlnMtch) and derive a Dual algorithm based on dual decomposition.
Dual allows us to make ItemRec decisions in real time, while satisfying long-term marketing constraints in expectation.
Finally, our connections between recommendation and marketing lead to novel applications.
We run experiments where we use marketing as an alternative to cold-start item exploration, by setting a positive minimal-exposure constraint for every item over the user-state distribution in a future period of time.
Our experiments are scalable to inﬁnite streams of user-states and open-sourced.
I NTRODUCTION
Many ML applications today involve decision making based on streams of events.
In recommender systems (RecSys), reasoning on the stream of events in a user’s past history has allowed real-time user-based item recommendations (ItemRec), leading to signiﬁcant impacts (Hidasi et al., 2016;
Hidasi & Karatzoglou, 2018; Ma et al., 2020; Chen et al., 2019; Kang & McAuley, 2018).
However, being user-centric, ItemRec has limitations in creating a globally inclusive environment for the item providers.
For example, ItemRec tends to over-expose a few popular items to improve the immediate user satisfaction, yet this causes the selected items to get even more popular, discouraging the other providers from creating seminal items with higher potential values in the future.
This is an important limitation because, as a RecSys involves the participation of both users and item providers, we must consider the success of the item providers for the long-term richness of the RecSys environment (Singh & Joachims, 2018; Mladenov et al., 2020; Su et al., 2021).
To cultivate the seed users for the new items, marketing via sponsored advertisements has been a practical strategy.
Note that we do not dismiss the possibility of uncontrolled marketing which hurts the utility of the users, but we argue that a minimal amount of controlled marketing in a trusted environment may actually yield better trade-offs between recommendation relevance and item-diversity.
For example, we may assign each item provider an exploration budget so that their items get exposed to a minimal percentage of users in a target period of time. (We treat two temporal states of the same user as two user(-state)s with separate marketing opportunities.) We further integrate marketing into [Table]
/Yb6ySvws1h44+u+NWApavuoTpzOUdKyqc/
Fv0Bq5x3XiuR/IFfnoFf9RMFHpqWkKkuM5ZKKOkEosVtNnYoCPG14s44sq7XssHxcbS8lZMXFwhaypj5dA1Wr1LCt5g8lXj6LVTF5MoqfjeJ3Twd7LxYns0HukwdkSGKyQ/bIa7JPxoSRD+QT+UK+Bp+Db8H34McZGvQWOffIBQt+/gHSsAWC</latexit> x = s ( t ) (a) Deﬁnition of the intensity model λ ( x, y ) between User-State x and Item y .
recommender systems to create an online matching (OnlnMtch) environment. Notice that OnlnMtch implies exposure planning for each item over all user opportunities in the target period of time. This yields an advantage over other exploration methods like contextual bandits (Li et al., 2010) and Thompson sampling (Agrawal & Goyal, 2013; Kawale et al., 2015), because it lowers the requirement for online updates.
As a result, OnlnMtch allows direct benchmarks on the relevance-diversity trade-offs using holdout back-test periods in real-world datasets, shown in Section 5. To make it happen, we contribute two algorithms: a RIM solution to the UserRec marketing problem on behalf of the item providers and a Dual algorithm for the OnlnMtch problem with its planning challenge. Both are realistic problems with their own applications, which we describe now.
First, in the marketing setup, our goal is to predict the relevance of each user with respect to the given item.
We validate the quality of our prediction by comparing our top-user recommendations (UserRec) with the set of users who truly interacted with the given item in the holdout back-test period.
See Figure 1 for illustration.
Besides direct ranking outputs, the raw prediction scores are also used in the OnlnMtch step. Related to cold-start item promotion, UserRec can be seen as an extension to targeted advertising, where the advertisers bid on search queries or related items to increase the exposure of their chosen items (Chang et al., 2020).
In our work, a search query or a related item may be seen as a single event – often the last event – in a user’s history.
By extending to the full histories of the actual users, we obviously gain more information, but also create a novel problem in the space of information retrieval, which is to compare the users as random time series instead of random variables.
At a high level, we encode the user time series into hidden states by reusing most of sequential ItemRec models (Cho et al., 2014; Hidasi & Karatzoglou, 2018; Ma et al., 2020; Wu et al., 2017;
Bai et al., 2019).
However, we cannot directly compare the user states conditioned on the provided item, which leads to a “Bayesian paradox” we show in Figure 2. Instead, we realize that our tests as well as real marketing campaigns often have implicit temporal sensitivity.
E.g., we must compare between a relevant but inactive user and an active but less relevant user, based on the likelihood of the desired user-item interactions in a near-future time period.
We introduce the intensity concept in temporal point processes (TPPs) (Reinhart, 2018; Rizoiu et al., 2017) to account for the novel time dimension. Correspondingly, we develop Recurrent Intensity Models (RIMs) to extend from recurrent ItemRec models. See Figure 1a for detailed deﬁnitions in intensity models.
Note that while TPPs have been widely discussed in the space of user-churn prediction, news trends or item dynamics, and physical events (Jia & Benson, 2019; Du et al., 2016; Leskovec et al., 2009; Chen et al., 2020; Hosseini et al., 2017; Wang et al., 2019a;b), we believe TPPs are relatively novel in the UserRec domain, which opens new venues for applications. Related, TPP-based UserRec can be used in social recommendation (Li et al., 2021). In Appendix A, we will discuss another example of UserRec where we aim to send email reminders to users who leave their online-shopping carts abandoned for a period of time (Lowe & Craig, 2021).
Different from likelihood-based UserRec, the focus there is on the causality between two event types: the reminder event and the checkout event. Our RIM framework can extend to this case, but we omit the discussion for now, because similar intervention signals may not be easily available in the most general cases. Our current discussion is analogous to implicit-feedback recommendation.
[Table] Now that they are trained, we consider other scenarios for RIMs to best utilize the real-time properties of these sequence models. Table 1 summarizes these new scenarios. Among them, OnlnMtch yields the best integration between recommendation and marketing.
On the other hand, it requires the distribution of materials in a future period of time, which we must plan ahead. We develop a Dual planner based on the empirical user-state distribution from a past period of time.
First, we set the desired exposure rate for every item as a constraint in the optimization of the global utility for all user-item matches.
While the item-exposure constraints couple all users in the primal form, thanks to optimization duality, they become independent slack variables in the dual form. As a result, the dual form allows real-time decisions just like the user-centric ItemRec, while meeting the item-exposure constraints in expectation over a similar user-state distribution in a future time period.
Our Dual algorithm is conceptually similar to (Huang et al., 2016; Mehta, 2013; Ding et al., 2019), but we use a different approach via entropic regularization similar to Sinkhorn’s algorithm (Cuturi, 2013). We found our variant to be easier to converge in our experiments.
The rest of the paper will show details of our RIM model, Dual planner, and experiments.
Additional discussions on related work can be found in the appendix.
R ECURRENT I NTENSITY M ODELING (RIM)
We are interested in modeling a temporal process represented by a series of event times and types of a given user.
Let S =
[( t j , y j ) : ∀ t 1 < t 2 < · · · < t j < . .
, ∀ y j ∈ Y ] to be a random process that generates user behaviors with a continuous time variable t and a type variable (i.e., the consumed item in the event) in a large-cardinal set y ∈ Y .
We consider a user-time as a unique instance variable X = S ( t ) = { ( t j , y j ) ∈ S : t j < t } , with observable user history until time t .
We use lower-case x = s ( t ) to indicate its realization and x ∈ X to be the set of all possible user-time instances in the recommendation problem.
In the ItemRec construct, the sequence models are called upon the arrival of a user event to predict the item choice by a conditional categorical distribution: [Equation] ) , where λ ( x, y ) = exp (cid:0) w [Equation]
( x [Equation] + b y + m ( x [Equation]
(cid:1)
[Equation]
where h ( · ) is a sequence encoder function, w y is a row in the decoder weight, which is also interpreted as the hidden embedding of a speciﬁc item, b y is a global bias in the item choices, and m ( x ) represents any extra terms that cannot be learned because they cancel between the nominator and the denominator.
The m ( x ) term reminds us that the raw ItemRec scores λ ( x, y ) cannot be used to compare across users. We aim to repurpose λ ( x, y ) for UserRec by associating m ( x ) with a user prior term that reﬂects their activeness in the RecSys, an important factor in UserRec. Timeless Bayesian versus temporal intensity Given our goal to select( x | y ) , our ﬁrst thought is to frame it by Bayes rule p ( x | y ) ∝ p ( y | x ) p ( x ) , i.e., relating m ( x ) to log p ( x ) and other normalization terms.
However, this leads to a paradox shown in Figure 2.
The paradox is because we represent a user as a timeless variable x , instead of a temporal point process (TPP) through the lens of s ( t ) .
TPP predicts when the next event will happen for a user in addition to what type it is.
TPP is different from other temporal models in that it predicts the event time as a moving target, whereas the other models aggregate time as a feature.
The temporal concept is best explained from a counting perspective.
Here, we model the absolute number of user-item interactions, ∆ N ( x, y ) , in a future time window, [ t, t + ∆ t ) , where s ( t ) coincides with x .
We adopt a Poisson assumption that the rate
Truncated [Figure] in last week ( x )
P ( y = D | x )
A,B 0.4 C 0.
∅ 0.3 ∅∅ (Bayesian) ×   p ( x =
[ A,B ])
= 0 .p
( x = C )
= 0 .p
( x = ∅ ) = 0 .⇒
  p ( x =
[ A,B ] | y = D ) ∝ 0 .
2 × 0 .
4 = 0 .p
( x = C | y = D ) ∝ 0 .
2 × 0 .
8 = 0 .p
( x = ∅| y = D ) ∝ 0 .
6 × 0 .
3 = 0 .(MTPP) ×
  λ ( x =
[ A,B ])
=
λ ( x = C )
=
λ ( x = ∅ ) =
µ ⇒   λ ( x = [ A,B ] | y
=
D )
∝ 2 × 0 .
4 = 0 .λ
( x = C | y
= D )
∝ 1 × 0 .
8 = 0 .λ
( x = ∅| y = D )
∝ µ × 0 .
3 = 0 .
3 µ
[Equation]
The deﬁnition of the intensity parameter is visualized in Figure 1.
Besides matching the expectations, we may further prescribe ∆ N ( x, y ) ∼ Poisson ( λ ( x, y )∆ t ) as a parametric distribution, especially in short time windows.
The Poisson log-likelihood is:
[Equation]
The intensity parameter connects three variables of our interests: user sequence x , item y and our prediction target ∆ N ( x, y ) in a future time window [ t, t + ∆ t ) .
Slicing ∆ N ( x, y ) in different directions yields different aspects of the problem.
Most importantly, we slice by ∆ t to connect it to the distribution of future event times.
We expect an active user to have higher intensities in unit time or, equivalently, shorter intervals between events.
Temporal Point Processes (TPPs)
We focus on the connection between the intensity parameter λ ( x ) =
λ ( t ; s ( t )) and the time t ∗ for the next interaction. (For this paragraph, we consider the interactions with any next items in general and omit the item sufﬁx.)
Intuitively, we split the time interval until the next event into a large set of inﬁnitesimal windows, i.e., [ t, t ∗ ) = ∪ τ [ τ, τ + d t ) , and insert Poisson likelihood with label n ( τ ) = 0 to all previous windows and n ( τ ∗ )
= 1 to the last window.
Plugging in equation 3 we get (cid:88)
[Equation]
Note that the d t inside the logarithmic term is moved to the left side when we convert the probability mass function to probability density function in the continuous variable T ∗ .
Reinhart (2018); Chen (2016) discuss rigorous derivations as well as additional properties.
Equation 4 is known as the inter-arrival time distribution, which directly formulates a loss function to learn the intensity parameters, without the need to explicitly bucketize time in equation 2.
The intensity functions can further be time-varying based on the user state x = s ( t ) .
The numerical integration may be solved by importance sampling (Mei & Eisner, 2017), ODE solvers (Chen et al., 2018), or analytical forms in the case of Hawkes processes, which we use in this paper for simplicity.
Marked-TPP Decomposition
While TPPs can be built for each event type (i.e., item choice) with cross-channel inﬂuences from other event types, this may lead to drastically different models than existing sequential ItemRec models.
Instead, we utilize a Marked-TPP decomposition,
[Equation]
where p ( y | x ) comes from a regular sequence model (1).
MTPP is the framework of our RIM proposals. It resembles Bayes rule and considers the user sequence x = s ( t ) as a point process.
E STIMATION OF THE I NTENSITY P ARAMETERS
While RIM is motivated to estimate the intensity of each user-item pair separately, we ﬁnd it more convenient to utilize existing sequential ItemRec models.
Based on (5), we keep the RNN (or Transformer) model for its categorical prediction of the preference direction , p ( y | x ) , while introducing separate estimators for the user-intensity prior , λ ( x ) .
RNN-Pop is our simplest model, where we use the length of the user histories in the training sets.
This approach works due to a homogeneous intensity assumption, where the number of past events directly correlates with the future event intensity.
It also assumes that all user activities are recorded over a comparable period of time, which often holds in practice, because RecSys often truncates the training sets by a ﬁxed time for scalability reasons.
RNN-Pop is used in the example in Figure 2.
RNN-Hawkes To extend the naive RNN-Pop models, we may break the homogeneity assumption to consider user state changes that affect future event intensity.
In this regard, Hawkes process assumes a positive stimulation through past user events and a gradual churn-out effect after a period of inactivity.
It models these effects via an exponentially-decaying kernel in the intensity parameters
[Equation]
, where φ
[Equation] , and µ, α, s > 0 .
Here, we extend it with a mixture of R latent kernels with learned coefﬁcients and learn it with tick software package Bacry et al.
We design the R kernels to have log-spaced half-lives between 10 − 3 and 10 t max where t max is the largest temporal span in user histories.
A large- s r kernel approximates the RNN-Pop model with a step function after each observation, whereas a small- s r kernel suggests a fast-diminishing effect, often within a short browsing session.
RNN-Hawkes-Poisson (RNN-HP)
For long prediction horizons, we may also calibrate Hawkes scores to eliminate the effects of fast-diminishing kernels.
In fact, the accurate long-horizon model,
[Equation]
takes the expectation with respect to stochastic integrals.
To calibrate for the long-terms intensity scores, we notice that the Hawkes states φ ( t ) are already positive.
We thus model the ﬁnal scores as
[Equation]
Our additive formulation is inspired by (Mei & Eisner, 2017) and modiﬁed by allocating the softplus to each coordinate to improve its numerical stability.
Another approach is to replace (6) with overdispersed distributions like negative binomial (Salinas et al., 2020), which we leave as future work.
Other Baseline Methods Practical personalized marketing models are often highly customized, e.g., with proﬁtability or seasonality concerns, and they are not always designed to cover all items in the catalog, which requires training labels for all user-item pairs, including user-item pairs with zero activities in a time period.
Despite so, we illustrate the general idea to use explicit recency, frequency, monetization (RFM) features for user segmentation (Fader et al., 2005).
Here, frequency and monetary spending in related items are correlated with a customer’s future life-time value (CLV) with respect to the given item and so is the time elapsed since last purchase with an inverse relation.
We build a model based on graph-convolutional feature aggregation similar to GCMC (Berg et al., 2017), followed by Bayesian-personalized ranking loss (BPR) (Rendle et al., 2009) against sampled negative users and items in the large space of all user-item pairs.
Notice that this GCMC model uses labels from an explicit time period, which is inferior to RIM because (a) it has fewer usable labels and (b) it bucketizes the labels instead of preserving their continuous time and order.
We attempt a mitigation we call GCMC-Ex that creates multiple periods to extract more labels and order information.
However, we do notice that perfecting this remedy leads to a temporal integration similar to our TPP derivation (4).
More details of this model can be found in Section B.
We also consider matrix factorization (MF) methods, such as ALS (Hu et al., 2008) and LogisticMF (Johnson, 2014) from Implicit package 2 and BPR (Rendle et al., 2009) from LightFM package (Kula, 2015).
Notably, LightFM-BPR is limited to sample either users or items, but not both.
We include our own BPR model for completeness.
Unlike RIM and GCMC, MF methods do not consider temporal features.
Require: approximately normalized λ ( · , · ) ∈ [0 , 1] with high probability; α ; β ; user-state distribution P ( X ) , often from a past period of time; step-size γ Ensure: ˆ v y ; recommend items for random user with real-time state x by TopK { λ xy − ˆ v y : y ∈ Y} init ˆ v y ∼ Unif( − 1 , 0) , ∀ y ∈ Y for k in [0 , 1 , . . , 100] do set (cid:15) = 0 .
8 k ; sample a batch of {+user-states X (cid:48) ∼ P ( X ) ; compute { λ xy : ∀ x ∈ X (cid:48) , y ∈ Y} ﬁnd root r x (ˆ π (cid:15) ( λ x : , u (cid:48) x , ˆ v )) = 
+}
for every x ∈ X (cid:48) given ˆ v ; project to u (cid:48)
x ≥ﬁnd root r y (ˆ π (cid:15) ( λ : y , u (cid:48) , v (cid:48) y ))
{+= 
 for every y ∈ Y given u (cid:48) ; project+} to v (cid:48) y ≤take step ˆ v y = (1 − γ )
ˆ v y + γv (cid:48) y , ∀ y ∈ Y  O NLINE M ATCHING AND D UAL P LANNING So far, we have learned RIM models as extensions from RNN/Transformers using MTPP intuitions.
They often yield better accuracy than the time-bucketized RFM baselines and timeless MF models. Further, RIMs yield greater impacts in the ofﬂine and online matching scenarios in Table 1. Both new scenarios can be formulated by a constrained optimization problem, which we describe now.
OnlnMtch as Constrained Optimization We start by reformulating ofﬂine UserRec by constrained optimization. Say we want to send marketing emails to Top-C users at a given time to promote Item y .
We represent the space of all user(-state)s at the given time as X (cid:51) x and rewrite λ xy = λ ( x, y ) for convenience.
We express the top-user recommendation as binary assignments π xy ∈ { 0 , 1 } , ∀ x ∈ X and it is easy to see that π xy can be equivalently solved by:
[Equation]
For online marketing or matching, we extend the user-state set to a draw from a distribution P ( X ) , which unions all user-states in a period of time by sampling every user at regular time intervals or based on their actual times of interactions, whichever is appropriate for the application. Each sample point is counted as an independent user-state instance.
We also replace the summation in (8) with Lebesgue integral in the probability space.
We then solve for the OnlnMtch assignments π xy ∈ { 0 , 1 } , ∀ x ∈ X , ∀ y ∈ Y with a linearly-relaxed optimization problem:
[Equation]
where we express the thresholds in relative terms, i.e., α = K / |Y| for Top-K ItemRec given a userstate x
and β (cid:48)
=
C / |X| for Top-C UserRec on behalf of an item y .
Unique to OnlnMtch, we can also insert a minimal-exposure constraint for each item with threshold β, ∀ y ∈ Y . Without loss of generality, we keep β and omit β (cid:48) to keep the discussions simple.
Additionally, deﬁne
[Equation]
For the OnlnMtch problem, our challenge is that the item constraints are measured over a distribution of users in a period of time, while OnlnMtch requires real-time recommendation / marketing decisions. This challenge can be met with optimization duality, which we describe next.
Dual algorithm We take ideas from optimization duality to rewrite the constraints in a Lagrangian form. I.e., equation 9 is equivalent to max
[Equation]
where (cid:26) Constraint r x ( π ) ≤ 0 holds; otherwise u x → + ∞ ⇒ L (cid:15) ( π, u, v ) →
−∞
Constraint r y ( π ) ≥ [-
 [ −-]0 holds; otherwise v y → −∞ ⇒ L (cid:15) ( π, u, v ) →
−∞
Notice that we also replace the 0 ≤ π ≤ 1 constraint with an entropic-trap regularizer H ( π ) = − π log( π ) − (1 − π ) log(1 − π ) . Our algorithm will ensure that π never escapes from this trap.
Next, we apply the standard trick to deﬁne a dual objective by switching the order of operations:
[Equation]
After some derivations (Lemma S1 in Section C.1), we ﬁnd an analytical solution to the inner-loop:
[Equation]
The solution intuitively suggests that the assignment of a user-item pair not only depends on their predicted intensity score, but also on the global dual variables u x and v y .
For example, if λ xy 1 > λ xy 2 for the majority of users while we intend to keep the global exposure rates for y 1 and y 2 to be equal, then we may choose v y 1 > v y 2 , causing the ﬁnal item-assignments to be reverted for some users to achieve the desired exposure outcomes.
Notice that the item-dual variables become nonpositive if our aim is to boost the item exposures to at least the predeﬁned β portion. Finally, we discuss the solution to the outer-loop, which also yields an intuitive algorithm.
To do so, we plug (13) to (12).
The derivations in Lemma S2 in Section C.1) lead us to a clean result:
[Equation]
, u, v y )) .
Setting both values to zero yields an iterative algorithm that alternates between solving u and v . We use simulated annealing from a larger initial (cid:15) > 0 to produce more stable results.
See Algorithmfor more details.
In the extreme case where (cid:15) → 0 , the two root-ﬁnding steps degenerate as (cid:26) Find u x = TopKThreshold( { λ xy − ˆ v y : y ∈ Y} ) for every x ∈ X ; project to u x ≥
Find v y =
Quantile( { λ xy − u x : x ∈ X} , β ) for every y ∈ Y ; project to v y ≤(15)
For larger (cid:15) , we use bisection search to solve for (14) in each step, which is possible because the gradient functions are monotonic with respect to the variables u x and v y , respectively.
The computational complexity is similar to ﬁnding quantiles by full sorts.
See Appendix C.2 for further discussions on the computational and storage complexity.
Connections to Online Bidding and Pacing. Real-time bidding is a common practice in the online advertising industry. The bidding price from an advertiser depends on the expected reward due to the placement, the estimated competitor-pricing distribution, as well as its pacing plan to manage the campaign budget over a period of time for best results.
In our Dual algorithm, we may see ﬂavors of all the components: the dual variable v y affects the overall ranking of the item in each user-matching opportunity, receives inﬂuences from all other items in each iteration of the alternating algorithm (15), and is calibrated against the empirical user-state distribution to achieve smooth pacing over time. Additionally, our Dual planner may extend to future time periods if we assume stationary user-state distribution, which we empirically observe, as well as dual-variable distribution for the other items, which we control.
Dual planner has direct applications in guaranteed display advertising, where advertisers buy contracts to show their ads to a guaranteed size of audience in a period of time (Bharadwaj et al., 2010).
Further, Huang et al.
; Mehta (2013); Ding et al. (2019) analyzed the online regrets in such guarantees with slightly different algorithms. These algorithms are designed to minimize online-regrets in adversarial settings. Comparatively, our work is under stochastic and stationary assumptions, but we may consider adjustments after short periods of time for more practicality.
Also, we found some difﬁculties in the convergence of their algorithms, potentially due to the skewed intensity-score distributions in our experiments. We leave the detailed comparisons for future work.
E XPERIMENTS
We conduct experiments in three RecSys datasets with unique properties.
• Movielens (ML) (Harper & Konstan, 2015) is a movie rating website that collects user preferences to study RecSys.
We use a small-scale ML-1M in an 2003 release with one million rating events between six thousand users and three thousand items to easily validate our ideas.
• Netﬂix (NF) 3 is a movie rental service where movie ratings from a user’s past affects their personalized ItemRec in the future.
NF is widely used in rating prediction and ItemRec, but less so in UserRec and OnlnMtch.
• Yoochoose (YC) 4 is a dataset used in RecSys 2015 Challenge with six months of activities for product recommendation such as general tools, toys cloths, electronics, and much more.
We highlight some data preparation steps.
Most of our RIM methods are self-supervised sequence models.
Thus, we only need to hold out test windows with equal horizons for the evaluation of
UserRec, ItemRec, and Mtch.
We use absolute time windows on NF and consider only the observed users/items in the training data as testing targets, because we may not know which new users/items will show up in the testing windows without temporal leakage from the test set.
Besides temporal splits, it is common for sequence models to split train/test sets by users.
We consider user-splits on
ML and YC, because most users are active at different, non-overlapping clock times.
We hold out time windows only on the test users (Group-B in Table S1), with equal starting time relative to their ﬁrst event as well as equal size or horizon.
All training users (Group A) and the observed histories of the testing users (Group B left part) are considered training data.
RNN-HP and GCMC(*) require further splitting of the training set.
On NF, we create a set-back window between [ T (cid:48) , T ) from all users and on ML/YC, we keep the same time [ T, T + ∆ T ) but change the user base to Group A for validation.
Similarly, to simulate OnlnMtch, we must calibrate
Online-Dual on different users or times to test the transferability of the user-state distributions in online scenarios.
We reuse the set-back window as RNN-HP and GCMC(*) for simplicity.
Statistics of our data splits are shown in Table S1.
5. U SER R EC AND I TEM R EC R ESULTS
We ﬁrst compare various models for the separate tasks of ItemRec and UserRec to validate our proposals in Section 3.
On the ItemRec dimension, we observe the usual story that user-conditional models outperform non-personalized models, but they are then topped by sequence models.
However, the comparison becomes interesting in the UserRec direction, where the adoption of raw probability scores in RNN and Transformers could lead to inferior results than the simple heuristic that recommends top users unconditionally of the given item, i.e., UserPopularity or the more advanced
TPP models for the marginal prediction, λ ( x ) .
This is because the user-conditional probability scores in RNN/Transformer fail to identify the active users with higher intensity priors.
On the other hand, with our RIM proposals, the sequence models become top in the UserRec direction as well.
For the baseline BPR methods, we also compare our implementation with open-source LightFM package.
LightFM only allows User/Item sampling, but not both at the same time.
In contrast, our version samples both sides concurrently, allowing a single model to perform well in both tasks.
We also compare GCMC with GCMC*, a variant that includes multiple time periods to extract more labels and relations.
We include similar plots for the precision metrics in the appendix.
[Figure] 5. O FFLINE AND O NLINE M ATCHING E XPERIMENTS
To validate the discussions in Section 4, we integrate UserRec into ItemRec by setting minimalexposure constraints.
We ﬁx α = 1% in equation 9 and vary 0 ≤ β ≤ 1% as item min-exposure constraints.
(Section F covers a more common marketing scenario where the user-to-item capacity ratio is greater than one, which leads to even larger margins in our RIM methods.
Here, we focus on the case with tighter constraints to highlight the subtleties and new possibilities in our Mtch setting.)
Choosing β
=
corresponds to the unconstrained ItemRec problem.
As we increase β , we observe an increase in item-exposure diversity, measured by perplexity
[Equation]
(16) where π ( y ) is proportional to the total number of times that y is recommended in front of a user.
Figure 4 compares the performance of various methods in three matching environments: Ofﬂinegreedy, Ofﬂine-Dual and Online-Dual.
Within each environment, we see that all methods exhibit trade-offs between ItemRec relevance and item diversity.
Comparatively, Transformer-Pop/HP yields best performance and compromises least relevance to achieve better diversity, which supports our claim that better modeling of user intensities leads to better welfare for both users and items.
Notice that showing diversity effects in ofﬂine experiments is a challenging task due to the vast amount of missing data for alternative treatment effects.
We attribute our success to the consideration of new problem dimensions.
We show similar results in Section F on other datasets and settings.
We see similar results across different environments in Figure 4(b)&(c).
Notably, the Online-Dual solver supports both real-time decisions and long-term exposure constraints over the distribution of all user-states in a future period of time.
The consistent performance across different scenarios demonstrates great practicality in the integration of ItemRec and UserRec in RecSys.
C ONCLUSION
RecSys that purely focuses on the immediate user satisfaction may hurt the overall diversity of the items being recommended. Instead, we study an exploratory marketing mechanism where each item will be guaranteed a minimal exposure rate to a suitable audience in a period of time.
In this way, we address the pain points in marketing and allow the item creators to focus their expertise on creation. We believe this improves the long-term diversity and liveliness of the RecSys.
We make two contributions.
Firstly, we turn a sequential ItemRec model around to recommend users on behalf of a given item.
Our challenge is to treat the users as time series instead of random variables.
We introduce novel connections to the intensity parameter of a temporal point process and develop a RIM solution that combines RNN (Transformer) and TPP. Our second contribution is to allow a RecSys to guarantee a minimal (maximal) exposure rate for every item that we intend to nourish (downsize) in a future time period. We call this scenario OnlnMtch and introduce Dual planner as a solution.
All of our methods learn by user mini-batches to achieve unlimited scalability in the size of the user-state set. Also, compared with other exploration methods that require online feedback, our methods have unique advantages in the ease of benchmarks using ofﬂine datasets.
We thank Vaibhav Sethi and Balakrishnan (Murali) Narayanaswamy for some early discussions on the idea to repurpose ItemRec models for UserRec applications. We thank Danielle Robinson and Yuyang (Bernie) Wang for their discussions on the topic of Neural-ODEs, which inspired us to look further into the literature around temporal point processes.
We appreciate the discussions with Lihong Li and Hsiang-Fu Yu on the general topic of targeted advertising.
We ﬁnally appreciate the anonymous reviewers for this and our earlier versions for their constructive comments and additional references.
A R ELATED W ORK AND D ISCUSSIONS
Non-sequential marketing strategies Marketing is a general area studied from multiple perspectives.
From a system’s perspective, our example of keyword and related-item bidding comes from close examination of large-scale extreme classiﬁer models (Chang et al., 2020).
The push notiﬁcation example considers explicit user states, which are commonly expressed via Recency, Frequency, and Monetary values (RFM)
(Fader et al., 2005).
By far, the most common and straightforward approach is to use Gradient Boosted Decision Trees to regress on the binary user-item engagement labels using these RFM features.
At a simpler level, without the manually constructed temporal features, the loss function is shared with traditional matrix factorization approaches, such as Alternating Least Squares (Hu et al., 2008), Bayesian Personalized Ranking (Rendle et al., 2009), and
Logistic Matrix Factorization (Johnson, 2014), all of which serve as UserRec baselines.
Neural temporal point processes We further receive inspirations from temporal point processes (Reinhart, 2018; Chen, 2016; Du et al., 2016; 2015),
Neural-ODEs (Chen et al., 2018), and their intersections (Jia & Benson, 2019; Chen et al., 2020; Mei & Eisner, 2017; Zuo et al., 2020).
Most of the works are great introduction of various aspects of TPPs and their modern development in GPU environments.
However, most of the works focus on the prediction for better temporal modeling without novel applications of the intensity parameters for different applications.
Along the line of better temporal models, (Jia & Benson, 2019) shows how Neural-ODE can be applied to a MTPP construction and (Boyd et al., 2020) focuses on the particular challenge that personalized user histories “exhibit large predictive uncertainty at the beginning of the sequences”.
We do observe that many users churn out shortly after initial engagement.
To model this temporal heterogeneity near the users’ beginning, we use a simple trick, where we add another related time series with exactly one event for every user to mark their start time.
Causality In our work, we consider item exposure as the primary factor and assume positive downstream effects afterwards.
We could also analyze the temporal correlation between an intervention event and the change in the user’s temporal intensity in another event type.
In the abandoned-cart example, let ˜ y be the email-reminder event at time ˜ t and y ∗ be the desired user-checkout event at time t ∗ .
We then deﬁne the intervention state of the user as ˜ x
= x ∪{ (˜ t, ˜ y ) } and model the temporal intensity of the subsequent checkout event by
[Equation] or its time-varying integral form similar to equation 4. Another difference to the likelihood-based intensity models in our main text is the inference routine. To choose which users to send the email reminders, we must also consider the users in the hypothetical intervention states.
I.e., we augment the user time series as ˜ x
= x ∪ { (˜ t, ˜ y ) } before computing their intensity scores in the effect items.
Besides the (Granger) causality research, temporal intervention effects can also help prevent bad consequences in marketing.
For example, we may combine the positive effects of a marketing email through likelihood-based intensity modeling as well as the risks of undesirable effects like user churn-out through causality models.
Some interesting work on temporal causality include (Xu et al., 2016; Li et al., 2018; Jiang & Li, 2016; Lundberg & Lee, 2017).
Convex optimization and online regrets On the long-term constraints, we reference from
Sinkhorn’s algorithm from optimal transport (Cuturi, 2013; Genevay et al., 2018) and the literature in long-term marketing via online campaigns (Huang et al., 2016; Jenatton et al., 2016;
Ding et al., 2019), with convergence analysis dating back to (Mehta, 2013).
Online convergence analysis further leads to novel applications such as online classiﬁcation with speciﬁcity constraints (Bernstein et al., 2010).
Diversity and Exploration
In our discussion of the Dual solution for OnlnMtch, we mention a dual solution in the form:
[Equation]
This solution is intrinsically connected to the idea of bandit exploration with upper-conﬁdence bound (UCB) (Li et al., 2010; Chu et al., 2011), where our adoption of the the minimal-exposure constraint leads to a positive boost to the raw prediction scores, i.e., − v ( y ) ≥ 0 .
Further exploration in this direction may lead to additional beneﬁts in the OnlnMtch paradigm.
B GCMC M ODEL D ETAILS
The basic idea in GCMC-CLV is to split user behavioral data in time to create a temporal-prediction scenario.
We hold out a targeting window from the observed user behaviors as CLV labels and build RFM features for every user at the start of the window.
Here, every user is represented by the aggregation of the items found in the user’s own history, which reﬂects the frequency of interactions in these items.
Due to the sparsity of the user-item pairs in the observed user histories, we use a graph convolution layer for the aggregation, which also extracts the semantic item embeddings similarly to graph-convolutional matrix factorization (GCMC).
As for recency, we construct a single user bias term based on the recency of the last purchase regardless of the item choices.
We omit the peritem recency terms due to their high (inverse-)correlation with the per-item frequency features.
We neither have monetary information in our experiments.
The main challenge in the scalable CLV approach is the vast amount of implicit-negative labels.
While we observe positive CLVs in the target window, we should also include every unobserved user-item pair as a zero CLV label.
This leads to an imbalanced label distribution, i.e., a signiﬁcant amount of zeros which we call negatives, and a small portion of non-zeros which we treat as positive (binary) labels.
To address the imbalance, we adopt a strategy of negative sampling to augment every positive user-item pair with negative users (˜ u ) and items (˜ ı ) , followed by sigmoid-triplet loss: loss = 0 .
5 (cid:16) E ˜ ı log(sigmoid(score( u, ı ) − score( u, ˜ ı )) + E ˜ u log(sigmoid(score( u, ı ) − score(˜ u, ı )) (cid:17)
Details of our negative sampling strategy is similar to Bayesian Pairwise-Ranking (BPR).
However, open-source BPR implementations are often limited to sampling only negative items, leaving negative users unconsidered and causing biases in the CLV predictions.
We thus include our implementation, with the additional beneﬁt of better integration with GCMC framework.
We sample the negative users (and items similarly) according to
[Equation]
where C ( u ) is the total number of appearances of the user (or item) in the training data.
Finally, the BPR scores are translation-invariant, because BPR only focuses on the relative comparisons instead of the absolute values.
This would be a problem when we port the scores for Dual, where we expect positive intensity scores.
We use softplus activation to preserve the larger values in the raw scores.
Our earlier attempt with sigmoid activation leads to saturation in the higher score values, which causes issues due to over-exploration.
C D UAL A LGORITHM : A DDITIONAL D ETAILS C. D ETAILED P ROOFS OF P RIMAL AND D UAL S OLUTIONS
Lemma S1. Show that the solution to max π L (cid:15) [Equation] is
[Equation] X , ∀ y ∈ Y . Proof. The proof is done by taking derivatives with respect to π xy , ∀ x ∈ X , ∀ y ∈ Y .
We ﬁrst expand out the Lagrangian form
[Equation] [Equation] Then, we take a derivative with respect to π xy as [Equation]
Setting the derivative to zero yields the desired solution
{+[Equation] Lemma S2. Show that the solution to min u ≥ 
 d (cid:15)+} ( u, v ) is in the roots of
[Equation]
projected to u ≥ 0 , ∀ x ∈ X . The other part of min v ≤ 0 d (cid:15) ( u, v ) can be similarly derived.
Proof. The proof is done by plugging in the primal solution from Lemma S1 to the Lagrangian form and then taking derivatives with respect to u x , ∀ x ∈ X . We ﬁrst rearrange the Lagrangian form:
[Equation] This simpliﬁes the substitution of the solutions from Lemma S1. Speciﬁcally, the underlined terms may cancel. Deﬁne z xy = λ
xy − u x − v y {+(cid:15) , such that ˆ π xy = e zxy 
+}
[-= 
 for every y ∈ Y given u ( x ) ; project if needed-]+ e zxy . The rest of the terms are
[Equation] Taking the derivative with respect to u x leads to
[Equation] Setting the gradients to zero leads to a solution to the unconstrained problem min u d (cid:15) ( u, v ) .
Additionally, since the gradients are monotone and independent with respect to u x , ∀ x ∈ X , we may also obtain a solution to the constrained problem in the space of u ≥ 0 by direct projections. C. C OMPLEXITY AND S TORAGE C OMPLEXITY
With a batch-size of B and an catalog size of |Y| = n , this leads to a temporary storage complexity of O ( Bn ) and a computational complexity of O ( Bn (log( n )+log( B ))) for each mini-batch.
Notice that to enable fast computations on streams of users, we must also avoid the storage of all user-item scores, but use a combination of sparse and low-rank matrices.
For example, we may delay the ﬁnal dot-product layer between the user and item representations in RNN/Transformer models, but instead just store the D -dimensional representations using as little as O (( B + n ) D ) storage.
The computational complexity for the dot-product layer is O ( BnD ) , which is comparable with sorting / bisection costs in the Dual algorithm.
In our experiments, we use 100 m/B mini-batches based on m total user states in empirical data.
D D ATA S TATISTICS AND O FFLINE P RECISION M ETRICS
Figure/Table S1 shows our data splitting strategies and resulting statistics in the three datasets we consider in Section 5.
With temporal split, we carve out a prediction time window between ( T, T + ∆ T ] and a calibration/validation window between ( T (cid:48) , T ] for time-bucketed models like
Hawkes-Poisson and GCMC and the observation of user-state distribution for Online-Dual simulation.
With user split, we pick Group B user to construct a test window between ( T, T + ∆ T ] .
[Figure]
Figure S1 shows the ofﬂine precision benchmark results which resemble our Figure 3 on recall metrics in Section 5.1.
The trends are similar.
We show recall in the main text because it caters more to user-welfare, whereas precision connects to the global objective in Mtch experiments.
Conversely, the precision numbers agree with the unconstrained optimization results in Figure 4,S2,S and Figure S4,S5,S6 for different combinations of tasks and datasets, respectively.
E I TEM R EC WITH M IN -E XPOSURE C ONSTRAINTS (NF AND YC)
Similar to the Netﬂix results in the main text, we show two more Mtch experiments on NF (Figure S2) and YC (Figure S3).
Comparing the Pareto fronts, we observe RNN-TPP variants > RNN > MF > Non-personalized models, which is consistent with our results on ML dataset [Figure] [Figure]
F U SER R EC WITH M AX -U SER -C APACITY C ONSTRAINTS
We consider 1% ≤ α ≤ 100% and β = 1% , both as upper-bound constraints in equation 9.
In the limit α = 100% , the problem becomes UserRec without the consideration of user capacities.
This yields best UserRec relevance, with a risk to overwhelm the users.
As α decreases, the users are expected to take a more balanced load of incoming items.
We use a similar perplexity metric to measure the user-load balances
[Figure]
[Figure]
[Figure]
Figure S4, S5, S6 show the trade-off between relevance and user diversity, which is an indirect result of capacity limitations per user.
From the curves, we ﬁnd larger advantages in the RNN-TPP variants.
This is because the problem is more focused on the advertising perspective, giving larger user-to-item capacity ratio, which highlights our RIM contributions.
The rightmost points in the ﬁrst two plots agree with the unconstrained ofﬂine UserRec precision metrics in Figure S1.
For online UserRec, we observe that most methods, including all of RIM and GCMC methods follow similar trends as ofﬂine UserRec (Figure S4.c, S5.c, and S6.c).
However, we do notice some artifacts in MF methods on ML and YC datasets.
The artifacts are related to the under-delivery of recommendations with respect to maximal user-capacity constraints, which leads to drops in the precision metrics.
For more details, we show the average percentage of users delivered per marketed item in Figure S7. The ideal percentage is 1% to agree with the β -constraint.
Notice that constraint dissatisfaction is a possibility in online ItemRec as well, but they do not affect our general observation of improvements in diversity-relevance trade-offs and we thus omit the discussions for simplicity.
Our discussions here extend to online ItemRec as well.
[Figure]
What is special in online UserRec is that the total number of recommendations is highly related to the satisfaction of the β -constraints.
For example, in the extreme case where α = 100% and β = 1% , the user capacities are entirely unconstrained, yet the total number of item exposures have to be controlled.
The other settings where 1% ≤ α ≤ 100% respect the capacity of each individual user, but still over-provision the overall user capacities for the desired number of item exposures.
In these cases, the β -constraints are observed by transferring the dual parameters, v ( y ) , from the calibration set to the test set through either temporal split (Figure S1.a) or user split (Figure S1.b).
I.e., Online-Dual does not use any user statistics in the test set, which are unavailable ahead of time in true online settings.
Despite so, the β -constraints are largely satisﬁed with RIM and GCMC methods in all datasets as well as MF methods on NF dataset with temporal splits.
The artifacts are only limited to MF methods on ML and YC datasets, both of which use user-splits.
Upon further inspection, we realize that the artifacts are due to the fact that we reuse a part of training data for calibration purposes, i.e., the [ T, T + ∆ T ) time window for Group-A users in Figure S1.b.
This causes Group-A users to have more training data, leading to higher recommendation scores during the learning of v ( y ) .
As a result, v ( y ) tends to be higher, causing over-penalization when they are transferred to Group-B users, who have less training data and generally lower recommendation scores.
This eventually translates to overall lower item-exposure rates compared with what is calibrated on the validation set, shown in Figure S7.
Notice that these artifacts are much less obvious on NF datasets, where the total training-data time (6 months) overwhelms the difference (15 days) between these two splits.
They are neither signiﬁcant in RIM / GCMC methods, where the user training data is averaged against the user time span, improving the stability of the user-state distribution.
In fact, the artifacts with MF methods reveal a side beneﬁt of our RIM (and GCMC) methods in our novel design of online experiments.