Generalized Delayed Feedback Model with Post-Click Information in Recommender Systems
Anonymous Author(s)
AffiliationAddress email
Abstract
Predicting conversion rate (e.g., the probability that a user will purchase an item) isa fundamental problem in machine learning based recommender systems.
However, accurate conversion labels are revealed after a long delay, which harms thetimeliness of recommender systems.
Previous literature concentrates on utilizingearly conversions to mitigate such a delayed feedback problem.
In this paper,we show that post-click user behaviors are also informative to conversion rateprediction and can be used to improve timeliness.
We propose a generalized delayed feedback model (GDFM) that unifies both post-click behaviors and earlyconversions as stochastic post-click information, which could be utilized to train
GDFM in a streaming manner efficiently.
Based on GDFM, we further establish anovel perspective that the performance gap introduced by delayed feedback canbe attributed to a temporal gap and a sampling gap.
Inspired by our analysis, wepropose to measure the quality of post-click information with a combination oftemporal distance and sample complexity.
The training objective is re-weightedaccordingly to highlight informative and timely signals.
We validate our analysison public datasets, and experimental performance confirms the effectiveness of ourmethod.
Introduction
Conversion rate (CVR) prediction has become a core problem in display advertising with theprevalence of the cost-per-conversion (CPA) payment model[1–3].
With CPA, advertisers bid forpredefined user behaviors such as purchases or downloads.
Similar demands also arise from onlineretailers that aim to improve sales volume[4], which need to recommend items with high conversionrates to users.
However, conversions (e.g., purchases) may happen after a long delay [5], which leadsto a delay in conversion labels.
The adverse impact of such a delayed feedback problem becomesincreasingly significant with a rapidly changing market, where the features of users and items areupdated every second.
Thus, how to timely update the CVR prediction model with delayed feedbackattracts much attention in recent years [5–11].
Since conversions are gradually revealed after click events, previous literature concentrates onutilizing available conversion labels that reveal earlier.
Inserting early conversions into the trainingdata stream will also introduce many fake negative samples [6], which will eventually convert buthave not converted yet.
To mitigate the adverse impact of fake negatives, Chapelle [5] proposes adelayed feedback model (DFM) to model conversion rate along with expected conversion delay, thenthe model is trained to maximize the likelihood of observed labels.
The delayed feedback modelis further improved by introducing kernel distribution estimation method [12] and more expressiveneural networks [10, 13, 14].
However, the need to maintain a long time scale offline dataset impedestraining efficiency of DFM based methods in large scale recommender systems[6].
To enable efficient training in a streaming manner, Ktena et al.
[6] proposes to insert a negative sample once a clicksample arrives and re-insert a duplicated positive sample once its conversion is observed.
The fakenegative samples are supposed to be corrected by an importance sampling[15] method.
Importancesampling based methods are further improved by adopting different sampling strategies [7–9, 16].
Besides the conversion labels, many events related with conversion exist in real-world recommendersystems[17–19].
For example, after clicking through an item, a user may decide to add this itemto a shopping cart.
Statistical data reveals that such post-click behaviors have a strong relationshipwith conversions: About 12% of items in a shopping cart will finally be bought, while the proportionis less than 2% without entering a shopping cart[17].
Intuitively, utilizing post-click behaviorscan potentially mitigate the delayed feedback problem in CVR prediction since the time delay ofpost-click behaviors is usually much shorter than conversions.
However, the corelations betweenpost-click behaviors and conversions are far more complex than only considering conversions, andthe problem is further complicated by the streaming nature of training with delayed feedback.
In this work, we propose a Generalized Delayed Feedback Model (GDFM) which generalizes thedelayed feedback model (DFM)[5] to unify post-click information and early conversions.
Based on
GDFM, we establish a novel view on learning with delayed feedback.
We argue that training usingpost-click information in the delayed feedback problem is intrinsically different from traditionallearning problems from three perspectives: ( i ) Estimating conversion rates via post-click actionsrequires more samples than using conversion labels directly, which highlights the importance ofsample complexity.
( ii )
The post-click actions bring information of past distributions, which incursa temporal gap.
( iii )
The signals provided by post-click actions are highly stochastic and lead tolarge variance on the target model during training, which may instead hinder the performance ofthe conversion rate prediction.
Inspired by our analysis, we propose to ( i ) measure the informationcarried by a post-click action with conditional entropy, which we empirically show is related to thesample complexity of estimating the conversion rate; ( ii ) measure the temporal distribution gap by time delay; ( iii ) stabilize streaming training with a regularizer trained on past data distribution.
We conduct various experiments on public datasets with delayed feedback under a streaming training and evaluation protocol.
Experimental results validate the effectiveness of our method.
To summarize,
• We propose a generalized delayed feedback model (GDFM) to support various user behaviorsbeyond conversion labels and arbitrary revealing times of user actions.
• From a novel perspective, we attribute the difficulty of efficiently utilizing user behaviors to
a sampling gap and a temporal gap in delayed feedback problems.
• Based on our analysis, we propose a re-weighting method that could selectively use the
user actions that are most informative to improve performance.
Our method achieves stableimprovements compared to baselines.
Background
[-Action1 Action
-]
In recommender systems, the data stream is collected from user behavior sequence.
Without lossof generality, we take an E-Commerce search engine as an example, while a similar procedure alsoholds for display advertising[2, 3] and personalized recommender[4].
When a user searches for akeyword, the search engine will provide several items for this user.
Here, a (user-keyword-item) tupleis called an impression , which corresponds to a sample x .
After viewing an item’s detail page, theuser may purchase it.
The probability that a user will buy after clicking is called conversion rate(CVR).
Conversion can be any desired user behavior, such as registering an account or downloadinga game.
Without loss of generality, we only consider one type of conversion in the rest of this paper,while our method can be applied to multi-class case[20] without modification.
If an impressionsample x is finally converted, it will be labeled as y = 1 and y = 0 otherwise.
The conversion rateof x corresponds to p ( y = 1 | x ) .
The distribution p ( x ) and p ( y | x ) changes rapidly in real-world recommender systems.
For example,when a promotion starts, the conversion rate may increase steeply for items with a large discount,and such promotion happens every day.
The CVR model has to be updated timely to capture suchdistribution change.
However, the ground-truth conversion labels are available only after a long delay:
Many users purchase several days after a click event x .
Usually, a sample will be labeled as negativeafter a long enough waiting time, e.g., 30 days[5], this delay time of conversion label is denoted as [Figure] δ y .
Besides conversion labels, post-click actions (behaviors) are denoted as a j , j ∈ { 1 , ..., m } .
Eachaction a j is paired with a revealing time δ j , which means the value of a j is determined after a δ jdelay.
For example, we can query the database to see whether a user has put an item into the shoppingcart 10 minutes ( δ j ) after clicking.
If true, this action is labeled as a j = 1 , and a j = 0 otherwise.
There can be several revealing times for a single type of action, e.g., we may choose to reveal theshopping cart information at 10 minutes, 30 minutes and 1 hour, they are treated as different actionssince the revealing time is different.
It is noteworthy that revealing the conversion labels is alsotreated as a type of action.
To capture the characteristic that the data distribution changes along withtime, the data distribution is denoted by p t ( x , y )
=
p t ( x ) p t ( y | x ) at time t .
Our goal is to predictp t ( y | x ) at time t by training a model q θ ( y | x ) , where θ denotes model parameters.
Without datadistribution shift, it is straightforward to optimize likelihood by sampling from p t .
However, in thedelayed feedback problem, the latest available labels have a δ y delay.
Thus, simply training usingavailable labeled data will lead to q θ ( y | x ) =
p t − δ y
( y | x ) instead.
Generalized Delayed Feedback Model
We propose the following generalized delayed feedback model to describe the relationship betweendata features x , conversion labels y , post-click actions a , revealing time of actions δ , and datadistribution at time t :
[Equation]
Our formulation unified previous delayed feedback models[5, 10–12] while enable more generalfeedback and revealing times.
p t ( y | x ) is the target CVR distribution, we assume that p t ( y | x ) isindependent to user actions a , which enables us to extend the existing CVR prediction frameworkseamlessly.
To model the post-actions, we introduce a post-action distribution p ( a | x , y , δ ) , whichdepends on sample features x , the conversion label y , and the revealing time δ .
We assume the postaction distribution p ( a | x , y , δ ) is fixed or changes much slower than p t ( y | x ) , so that the post-actiondistribution does not depend on time t .
This assumption is critical for learning under delayed feedback since the relationship between post-actions and conversion should be stable so that post-actionscan be informative.
Previous literature also relies on similar assumptions implicitly, for example,a predictable delay time of conversion[5] or a loss function that depends on a stable post-actiondistribution in our formulation[7–9].
Our formulation decouples the post-action distribution andthe conversion distribution, which enables us to formulate this assumption explicitly and conductfurther analysis based on this assumption.
An alternative but more intuitive definition is to definep t ( x , y , a , δ ) =
p t ( a | x , δ ) p ( y | x , a , δ ) p ( δ ) p ( x ) , which requires to sum out a and δ to make a prediction.
We discuss this approach in the supplementary material.
We further introduce a revealingtime distribution of post-actions p ( δ ) , which is independent of other variables.
δ is also known as the elapsed time in previous literature [5, 8].
The revealing time enables us to inject action information atcontrollable time δ , which is typically less than δ y so that the model can be updated much earlier.
Weintroduce a streaming training method of GDFM in the next section.
3.1 Training GDFM
We introduce an action prediction model q ϕ ( a | x , y , δ ) to estimate the action distribution p ( a | x , y , δ ) ,the CVR prediction model is denoted as q θ ( y | x ) , which will be used to predict the conversion ratep t ( y | x ) .
In the data stream, not all the information in GDFM defined in Eq.
(1) is available atthat same time, so we propose a streaming training method for GDFM that can utilize availableinformation in the data stream.
According to the availability of a , y , there are several different cases.
We assume the current timestamp is t .
When a , x , y is available The action prediction model q ϕ ( a | x , y , δ ) can be trained with followingloss when a , x and y are available.
[Equation]
Note that with the assumption that p ( a j | x , y , δ j ) is invariant with respect to t , loss Eq.
(2) will beminimized when p ( a j | x , y , δ j ) =
q ϕ ( a j | x , y , δ j ) .
When y , x is available When we have the ground-truth label of y , we are able to update the CVRmodel directly by minimizing cross entropy.
[Equation]
Note that this loss is minimized when p t − δ y
( y | x ) =
q θ [Equation] δy ( y | x ) [Equation]
When a , x is available Since the delay times of user actions ( δ j ) are smaller than the conversionlabel delay δ y , we can update the CVR model once a user action is observed.
When we observe thej th user action a j at revealing time δ j , we can update the prediction model q θ ( y | x ) using followingproxy delayed feedback loss :
[Equation]
which is depicted in Figure.
1 (a).
However, it is unclear whether minimizing Eq.
(4) will helplearning p t ( x | y ) .
3.2 Narrowing the delayed feedback gap with post-actions
The training objective of the proxy feedback loss can be viewed as a multi-task training problem withdifferent actions, which is also explored by Hou et al.
[10] and Li et al.
However, learning withdelayed feedback is different from general multi-task learning problems without delayed feedback.
In learning with delayed feedback, since we are training on different data distribution at differenttimes, the tasks are intrinsically conflicted, which may be harmful to the target [10, 21].
We analyzeits effect in the following sections.
3.2.1 Post-actions can be informative
Since in GDFM we are using post-actions from t − δ j instead of t − δ y , we have a chance to dobetter.
But will optimizing Eq.
(4) improve the performance of q ( y | x ) ?
Without loss of generality,we consider an action a ∈ { a 1 , ..., a k } , and p ( a | x ) = (cid:80) y p ( y | x ) p ( a | x , y ) (condition on δ omitted),y ∈ { y 1 , ..., y n } .
Assuming we can estimate p ( a | x ) accurately and k ≥ n , we have following results:
Lemma 3.1.
Use a matrix M x ∈ R k × n to denote the conditional probability p ( a | x , y ) , where( M x ) ji = p ( a = a j | x , y = i ) .
We can recover p ( y | x ) from p ( a | x ) if and only if rank( M x ) = n .
Proof sketch : p ( a | x ) is a linear transformation from p ( y | x ) and the transform matrix is given by
M x .
By construction, a solution exists and rank( M x ) = n guarantees that the solution is unique.
Lemma.
3.1 highlights the benefits of utilizing post-actions: if the relationship between a postaction and the target is predictable (we can train a model q ( a | x , y ) to approximate p ( a | x , y ) ) and informative ( rank( M ) = n ) we can recover the target distribution even without the ground-truthlabel y .
Since rank( M ) < n requires that some rows of M are linear combination of the others,which rarely happens in real-world problems, we assume the assumption holds true in the next section.
We discuss the case that this assumption fails in Section.
However, even if rank( M ) = n ,recovering p ( y | x ) is still challenging.
Because we are estimating p ( a | x ) using samples from p ( a | x ) ,and the difficulty of estimating p ( y | x ) via estimating p ( a | x ) also depends on properties of p ( a | x , y ) .
3.2.2 Measuring information carried by actions
Not all actions carry the same amount of information about p ( y | x ) .
For example, an action maycarry zero information about y if it is constant and irrelevant to y ; it may carry full information if italways equals y .
The general relationship between a and y is far more complex, which lies betweennon-informative and fully informative.
We want to estimate the information quantity of each actionto utilize actions more efficiently.
To this end, we propose to use conditional entropy to measureinformation carried by actions.
The conditional entropy of y given a can by calculated by
[Equation]
Conditional entropy measures the amount of information needed to describe y given the value ofa .
When a is non-informative to y , that is p ( a , y ) =
p ( a ) p ( y ) , we have H ( y | a ) =
H ( y ) , which isthe maximum value of H ( y | a ) ; when a determines value of y , that is, there is a function y =
f ( a ) ,we have H ( y | a ) = 0 .
Since the scale of H ( y | a ) only depends on p ( y ) , this information metric iscomparable among different a , which is a desired property.0.
[Figure]
Empirically, we found that conditional entropy is also relatedto the sample complexity of estimating p ( y | x ) via p ( a | x ) .
Inlearning with delayed feedback, we are sampling from differentdistribution p t at different time stamp t , which indicates that toperform well, we should be able to learn fast.
Thus, analysisthat is based on a large amount of i.i.d samples from a fixeddistribution can’t capture the difficulty of learning with delayedfeedback.
Property testing [22] considers sample complexity oflearning distributions.
Specifically, distinguishing between twodiscrete distributions p and q with a probability at least 1 − ϵrequires Ω( 1 D 2 TV ( p,q ) log 1 ϵ ) samples from q and p .
Where D TVis the total variance distance which is defined by D TV ( p, q ) =
193 (cid:80) i | p
( i ) − q
( i ) | .
Since we are estimating p ( y | x ) through aproxy distribution p ( a | y ) , the sample complexity depends onhow close the transformed distribution p ( a | x ) is.
Empirically,we found that using p ( a | x ) will make p ( y | x ) more difficultto learn, that is, the distance of two distribution p 1 ( y | x ) and p 2 ( y | x ) will decrease when theyare transformed to p 1 ( a | x ) and p 2 ( a | x ) correspondingly.
So the difficulty of learning p ( y | x ) viap ( a | x ) depends on the change of distribution distance incurred by the stochastic transformationp ( a | y ) .
We empirically investigate the relationship between the change of distribution distance andconditional entropy by Monte Carlo sampling, the results in Figure.
2 indicate that as the conditionalentropy increases, the transformed distribution distance decreases exponentially, which motivates thefollowing information weights
[Equation]
where α is a hyper-parameter.
Eq.
(6) roughly measures the reciprocal of sample complexity oflearning p t − δ j
( y | x ) via p t − δ j ( a j | x ) , so that actions are weighted based on their effective informationcarried by each sample.
3.2.3 Measuring temporal gap
Besides the sample complexity of estimating p t − δ j ( y | x ) via action distribution, the intrinsic temporalgap between p t and p t − δ j also influence the predicting performance.
Even if we have unlimitedsamples from p t − δ j we are only able to recover p t − δ j ( y | x ) instead of p t ( y | x ) .
So we also need totake the revealing time δ j into consideration.
However, we can’t estimate the gap between p t − δ j and p t accurately since the exact distribution is unknown and changes along with time.
Empirically(Figure.
3) we found that the gap measured by KL divergence tends to increase as the time gapincreases.
So we propose the following temporal weight
[Equation]
which measures the gap between p t and p t − δ j .
β is a hyper-parameter that reflects the expectedchanging speed of data distribution along with time, larger value of β indicates that we discard moreinformation from stale data.
Overall, we introduce a weight on loss Eq.
(4) as follows:
[Equation]
For simplicity, our w j does not depend on x , the extension to x dependent weights is straightforward.
The weight w j can be decomposed into two components, the first component w info measures howinformative action a j is to the target y .
The second component w time measures the gap between p tand p t − δ j .
The procedure of calculating weight vector w ∈ R m is summarized in Algorithm.
1: for { x , y , a , δ } in data stream do 2: for { x , y , a } that y is unlabeled do 3:
Update q θ ( y | x ) with Eq.
4: end for 5: for { x , y , a } that y is labeled do 6:
Update q ϕ ( a | x , y , δ ) with Eq.
Update q θ t − δy ( y | x ) with Eq.
8: end for 9: end for
Input : Dataset { y , a }
Parameter : α , β .
1: Estimate p ( a , y ) by counting.
2: Calculate H ( y | a ) with Eq.
3: Calculate w info with Eq.
4: Calculate w time with Eq.
3.2.4 Reducing variance by delayed regularizer
Estimating probability with sampling will incur instable variance during stream training, which mayharm performance.
Since we are estimating using a stochastic proxy p ( a | x ) , the variance will beeven higher.
To reduce variance and stabilize training, we introduce a regularizer loss Eq.
(9) thatconstraints update step during training, where q θ t − δy ( y | x ) is trained with loss Eq.
[Equation]
Besides reducing variance during training, another critical function of L reg is to make GDFM safeto introduce more actions.
The analysis in Section.
3.2.1 shows that if an action is informative,we can learn p ( y | x ) through p ( a | x ) .
However, if rank( M ) < n (typically because k < n ) or theinformation carried by p ( a | x ) is too weak to recover p ( y | x ) as analyzed in Section.
3.2.2, GDFMmay fail to learn p ( y | x ) because of interference from action signals.
The regularizer loss L regintroduces the ground-truth labels of y from p t − δ y , which guarantees that the performance learned by
GDFM will not be much worse than p t −
δ y
(the Vanilla method in experiments).
In this perspective,
L reg make it safer to introduce various post-click information into our training pipeline withoutworrying about sudden performance drop, which is critical in production environments.
The overallloss function with revealing time δ j is
[Equation]
The streaming training procedure of GDFM is summarized in Algorithm.
Experiments
4.1 Datasets and data analysis
We use two large-scale real-world datasets: 1) Criteo Conversion Logs 1 is collected from an onlinedisplay advertising service within 60 days and consists of about 16 million samples with conversion labels and timestamps[5]. 2) Taobao User Behavior 2 is a subset of user behaviors on Taobaocollected within 9 days and consists of more than 70 million samples and 1 million users[23].
Taobaodataset provides user behaviors within (page-view, buy, cart, favorite).
We conduct data analysis on the datasets to validate our assumption on GDFM.
The results aredepicted in Figure.
First, we investigate the change of conversion rate distribution p ( y | x ) .
Sinceexact p
( y | x ) is not available, we train a CVR model on the first 30 days to estimate the conversionrate at the 30th day, which is denoted as p
( y | x ) .
Then we fine tune this model day by day to estimatethe conversion rate at p t ( y | x ) , here t denotes the t th day.
In this way, we can estimate the temporalgap between p 0 ( y | x ) and p t ( y | x ) by their Kullback-Leibler divergence D KL ( p 0 || p t ) .
In Figure.
3(Left) we can see that the temporal gap grows along with time, which necessitate the analysis of timedependent conversion distribution p t ( y | x ) in Section.
Secondly, we investigate the informationquality of actions with conditional entropy as proposed in Section.
Specifically, we use theobserved conversion at revealing time δ as post-action A δ , and vary the revealing time from 0 to 30days.
We can infer from Figure.
3 (Left) that the information carried by this action steadily increase(the conditional entropy decrease) along with time, which is intuitive that the latter actions are moreinformative.
Thirdly, we investigate the relationship between post-click actions and conversions.
Weplot the conversion rate and the proportion of cart and favorite actions in converted samples (whichcorresponds to p ( a = 1 | y = 1) ) in the Taobao dataset in Figure.
3 (Right).
We can see that the actiondistribution p ( a = 1 | y = 1) is stabler comparing to conversion rate, which conforms our assumptionof stable action distribution in Section.
4.2 Evaluation
Streaming evaluation We follow a streaming evaluation protocol proposed by Yang et al.
Specifically, the datasets are split into pretraining and streaming datasets.
An initial conversionrate prediction model as well as auxiliary models (e.g., the importance weighting model in ESDFMand action distribution model q ϕ in GDFM) are trained on this pretraining dataset to simulate astable state after a long time of streaming training in real-world recommender systems.
Then themodels are evaluated and updated hour by hour in the streaming dataset.
Each method is trainedwith the available information at the corresponding timestamps.
Following practice in [8–10, 16],we report the receiver operating characteristic curve (AUC), precision-recall curve (PR-AUC) andnegative log-likelihood (NLL).
The metrics are calculated within each hour.
We report the averageperformance on the streaming dataset.
Baseline methods To evaluate the performance of GDFM, we compare with the following methods:
1) Pretrain :
The CVR model is trained on the pretraining dataset, then fixed in streaming evaluation.
The following methods start with this pre-trained model at the beginning of the streaming evaluation.
) Vanilla : Wait for δ y time, then use
conversion
labels to
train the model
3) Oracle Use conversionlabels
to train without delay.
The oracle method corresponds to the upper bound of performancewith feedback delay.
We compare two representative importance sampling methods. 4) FNW [6]: A sample is labeled as negative on arrival, and a duplicate is inserted once its conversion is observed.
The fake negative weighted (FNW) loss adjusts the loss function; 5) ES-DFM [8]: After waiting fora predefined time, a sample is labeled as negative if conversion has not been observed.
If a sampleconverts after the waiting time, a duplicate is inserted.
The loss function is adjusted by the ES-DFMloss.
We also compare with a multi-task learning method based on DFM.
6) MM-DFM [10]: Treatingpredicting the observed conversion label as a multi-task learning problem, the tasks are optimizedjointly using streaming data.
7) Linear Relation [24]: Modeling the relationship between user actionswith linear layers, inspired by Chen et al.
[Table]
Implementation Following [8, 9], we discretize and treat numerical features the same as categoricalfeatures in the Criteo dataset.
Since the Taobao dataset does not provide user features, we use thelast 5 user behaviors as features[23].
Inspired by Weinberger et al.
[25], we hash user ID and item
ID into bins and use an embedding to represent each bin.
We use the same architecture for all themethods to ensure a fair comparison.
All the methods are carefully tuned.
We use α = 2 , β = 1 ,λ = 0 .
01 , lr = 10 − 3 for GDFM.
Because of the page limit, the details of implementation and codeare provided in the supplementary material.
The performance is reported in Table.
Following [8, 9, 16], we report the relative improvementof each method to the performance gap between the Pre-trained model and the Oracle model.
Fromthe results, we can infer that, 1) GDFM performs significantly better than compared methods. 2)
It isnoteworthy that FNW and ES-DFM brings a significant negative impact to NLL on the Taobao dataset,which is caused by the fake negative samples introduced by them; 3)
The performance of FNWand MM-DFM has a more considerable variance compared with ES-DFM and GDFM (especially
NLL on Taobao), since ES-DFM uses a fixed weighting model to re-weight its loss, which has asimilar effect to GDFM’s explicit regularizer that can stabilize training; 4) On the Criteo dataset,
GDFM outperforms other methods without using post-click information other than conversion labels,which indicates that our information measure is effective for early conversions; 5) On the Taobaodataset, GDFM also utilizes post-click information such as cart and favorite.
The results indicate thatintroducing rich post-click information into GDFM can improve the performance of CVR prediction.
4.3 Experimental analysis of GDFM
To further investigate the source of performance gain of GDFM, we construct several experiments.
First, we remove the effect of information weights and the regularizer loss by setting the coefficientsto zeros correspondingly.
The results reported in Table.
(2) indicate that the performance of GDFMis a combined effect of different components.
Without α and λ , the performance drops significantly,which indicates that sample complexity and training variance are influential factors in the Criteodataset.
Dropping β has relatively little influence to performance, which indicates that the temporalgap in the Criteo dataset has a weaker influence.
Secondly, to validate the capacity of GDFM to deal with more actions, we construct an additionaluser action in the Criteo dataset.
This action is set to the same as y with a probability p , andset to 1 − y with probability 1 − p .
In this way, we can investigate how the information measure influences the performance of GDFM.
We report the results with different p in Table. [Table] [Table]
When p = 0 .
5 , the newly inserted action does not contain information about y , and the performance does not decrease, which indicates that GDFM can extend to more actions safely; Whenp > 0 .
5 , the action becomes more informative, and the performance of GDFM steadily improves,
[Figure]
which indicates that GDFM can utilize the informationcarried by actions efficiently;
When p = 0 .
95 , the actionis highly informative to y , the performance approaches
Oracle, which indicates that GDFM can efficiently extractinformation from high-quality action features.
Thirdly, we plot the weigths w info , w time and w corresponding to different revealing time δ , with α = 2 , β = 1 asin other experiments.
We can infer from the Figure.
4that 1) the information carried by actions increases as therevealing time increases; 2) the temporal gap enlarges asthe revealing time increases; 3) the combined information weights w reaches its maximum value on the 7th day,which indicates that the information revealed on the 7thday is most informative to conversion prediction.
Information revealed at other times still has non-negligible weightsthat influence training.
Related work and discussion
Delayed feedback model (DFM) [5] firstly introduces survival analysis[26, 27] to deal with thedelayed feedback problem.
The following methods predict the probability of whether a user convertsbefore a predefined waiting time[8–10].
However, how to define the strategy of duplicating trainingsamples varies among previous literatures[8–10, 16].
On the contrary, the revealing time distributionis explicitly defined in GDFM, which indicates that each sample should be duplicated once at arevealing time.
The information weight w in GDFM is effectively adjusting the revealing time withthe aid of GDFM’s information measure.
To the best of our knowledge, we are the first to analyze the property of learning with delayedfeedback with a time-varying distribution.
Existing analysis[5, 8, 9, 16] assumes the distributionis stable (unlimited samples from p ( x , y ) )
[5], or the action distribution p ( a | x ) can be estimatedaccurately[8, 9, 16].
Since p t ( x , y ) indeed varies along with time (otherwise streaming training isunnecessary), assuming a static p ( x , y ) can not capture the property of the delayed feedback problem;
The action distribution p ( a | x ) =
(cid:80) y p ( a | x , y ) p ( y | x ) also changes following p t ( y | x ) .
Thus, GDFMcan be a better model for understanding and analyzing the delayed feedback problem.
Delayed feedback problem has also attracted attention from bandit learning communities[28–30],where the objective is to minimize regret in a decision problem.
Vernade et al.
[28] introduces a similaridea of using intermediate observations before getting rewards and shows that such intermediateobservations can improve regret, which agrees with our analysis from a different perspective.
Learning with heterogeneous feedback aims to utilize different user actions jointly. Chen et al. [24]proposes to predict conversions as a linear transformation of the CTR prediction; Chen et al.
[31] and
Jin et al. [32] further model the relationship with graph neural networks.
However, these methods cannot tackle changing labels in the delayed feedback problem. A limitation of GDFM is that the training cost grows linearly with the number of different revealingtime δ j , which may be a potential bottleneck in large-scale streaming training.
Checklist
For all authors...
(a) Do the main claims made in the
abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] see Section.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See supplementary.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code and detailed information in supplementary.
Data urls at footnote 2, 1.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Details in supplementary.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Details in supplementary.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Licenses are in the urls.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board approvals, if
[Table]
[Table] [Table] [Table]
amountspent
N [Table]