  One of the difficulties of conversion rate (CVR) prediction is that the
conversions can delay and take place long after the clicks. The delayed
feedback poses a challenge: fresh data are beneficial to continuous training
but may not have complete label information at the time they are ingested into
the training pipeline. To balance model freshness and label certainty, previous
methods set a short waiting window or even do not wait for the conversion
signal. If conversion happens outside the waiting window, this sample will be
duplicated and ingested into the training pipeline with a positive label.
However, these methods have some issues. First, they assume the observed
feature distribution remains the same as the actual distribution. But this
assumption does not hold due to the ingestion of duplicated samples. Second,
the certainty of the conversion action only comes from the positives. But the
positives are scarce as conversions are sparse in commercial systems. These
issues induce bias during the modeling of delayed feedback. In this paper, we
propose DElayed FEedback modeling with Real negatives (DEFER) method to address
these issues. The proposed method ingests real negative samples into the
training pipeline. The ingestion of real negatives ensures the observed feature
distribution is equivalent to the actual distribution, thus reducing the bias.
The ingestion of real negatives also brings more certainty information of the
conversion. To correct the distribution shift, DEFER employs importance
sampling to weigh the loss function. Experimental results on industrial
datasets validate the superiority of DEFER. DEFER have been deployed in the
display advertising system of Alibaba, obtaining over 6.0% improvement on CVR
in several scenarios. The code and data in this paper are now open-sourced
{https://github.com/gusuperstar/defer.git}.
