U NCERTAINTY - AWARE OFF POLICY LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines, recommender systems, etc.
While the ground-truth logging policy, which generates the logged data, is usually unknown, previous work directly takes its estimated value in off-policy learning, resulting in a biased estimator.
This estimator has both high bias and variance on samples with small and inaccurate estimated logging probabilities.
In this work, we explicitly model the uncertainty in the estimated logging policy and propose a novel Uncertaintyaware I nverse Propensity Score estimator (UIPS) for improved off-policy learning.
Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
I NTRODUCTION
In many real-world applications, including search engines (Agarwal et al.
online advertisements (Strehl et al. (2010)), recommender systems (Chen et al.
Liu et al.
(2022)), only logged feedback data is available for subsequent policy optimization.
For example, in recommender systems, various complex recommendation models (i.e., policies) (Zhou et al.
; Guo et al.
(2017)) were optimized with logged user interactions (e.g., clicks or staytime) to items recommended by previous recommendation policies.
However, such logged data is known to be biased, since one does not know the feedback on items that previous policy (which is generally referred as the logging policy ) did not take.
This inevitably distorts the evaluation and optimization of a new policy when it tends to select items that are not in the logged data.
Off-policy learning (Thrun & Littman (2000); Precup (2000)) emerges as a favorable way to learn an improved policy only from the logged data by addressing the mismatch between the learning policy and the logging policy.
One of the most commonly used off-policy learning methods is Inverse Propensity Scoring (IPS) (Chen et al.
; Munos et al.
(2016)), which assigns per-sample importance weight to the training objective on the logged data, so as to get an unbiased optimization objective in expectation.
The importance weight in IPS is the probability ratio between the learning policy and the logging policy.
However, the ground-truth logging policy is unavailable to the learner, e.g., it is not recorded in the data.
One common treatment taken by previous work (Strehl et al.
; Liu et al.
; Chen et al.
; Ma et al.
(2020)) is to first employ a supervised learning method (e.g., logistic regression, neural networks, etc.) to estimate the logging policy, and then take the estimated logging policy for off-policy learning.
We theoretically show that such an approximation results in a biased estimator which is sensitive to those inaccurate and small estimated logging probabilities.
Worse still, the small values of the estimated logging probabilities usually mean that there are fewer related samples in the logged data, so its estimation usually has high uncertainties, i.e., inaccurate estimation with high probability.
Figure 1 shows a piece of empirical evidence from a large-scale recommendation benchmark KuaiRec dataset (Gao et al.
(2022)), where items with lower frequencies in the logged dataset have lower estimated logging probabilities and higher uncertainties concurrently.
The high bias and variance caused by these samples greatly hinder the performance of off-policy learning.
In this work, we explicitly take the uncertainty of the estimated logging policy into consideration and design a novel Ucertainty-aware Inverse Propensity Score estimator (UIPS) as the optimization objective for policy learning.
UIPS introduces an additional weight to approach the ground-truth propensity from the estimated one, and learns an improved policy by alternating: (1) Find the optimal weight that makes the estimator as accurate as possible, taking into consideration the uncertainty
· 10 −
Log(frequency)
E s t i m a t e d L ogg i n g P r o b a b ili t y (a)
{+Estimated Logging Probability 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 0 .2 .4 .6 .8 . 
+}
· 10 −
Log(item frequency)
U n c e r t a i n t y
[Figure]
of the estimated logging policy; (2) Improve the policy by optimizing the resulting estimator.
We further find a closed-form solution for the optimal weight by deriving an upper bound on the mean squared error (MSE) to the ground-truth policy value.
The optimal weight adjusts sample weights considering both the uncertainty of estimated logging probabilities and the propensity scores, rather than simply boosting or penalizing samples with high uncertain logging probabilities.
Experiment results on the synthetic and three real-world recommendation datasets demonstrate the efficiency of
UIPS.
All data and code can be found in supplementary materials for reproducibility.
To summarize, our contribution in this work is as follows:
• We point out that directly using the estimated logging policy leads to sub-optimal off-policy learning, since the resulting biased estimator is greatly distorted by samples with inaccurate and small estimated logging probabilities.
• We take the uncertainty of the estimated logging policy into consideration and propose UIPS for more accurate off-policy learning.
• Experiments on synthetic and three real-world recommendation datasets demonstrate UIPS’s strong advantage over state-of-the-art methods.
P RELIMINARY : OFF - POLICY L EARNING
We focus on the standard contextual bandit setup to explain the key concepts.
Following convention (Joachims et al.
; Saito & Joachims (2022);
Su et al.
, let x ∈ X ⊆
R
d be a ddimensional context vector drawn from an unknown probability distribution p ( x ) .
Each context is associated with a finite set of actions denoted by A , where |A| < ∞ .
Let π : A×X
→
[0 , 1] denote a stochastic policy, such that π ( a | x ) is the probability of selecting action a under context x and (cid:80) a ∈A π ( a | x ) = 1 .
Under a given context, reward r x ,a is observed when action a is chosen.
Take news recommendation for example, x represents the state of a user, summarizing his/her interaction history with the recommender system, each action
a is a candidate news article, the policy is a recommendation algorithm, and the reward r x ,a denotes the user feedback on article a , e.g., whether the user clicks the article.
Let V ( π ) denote the expected reward or value of the policy π :
[Equation]
We look for a policy π ( a | x ) to maximize V ( π ) .
In the rest of the paper, we denote E x ∼ p ( x ) ,a ∼ π ( a | x ) [ · ] as E π [ · ] for simplicity.
In contrast to performing online updates by following the learning policy π ( a | x ) , in off-policy learning we can only access a set of logged feedback data denoted by D := { ( x n , a n , r x n ,a n ) | n ∈ [ N ] } , where [ N ] := { 1 , . . .
, N } .
Given x n , the action a n was generated by a stochastic logging policy β ∗ , i.e., the probability action a n was selected is β ∗ ( a n | x n ) .
The actions { a 1 , . . . , a N } and their corresponding rewards { r x 1 ,a 1 , . . . , r x N ,a N } are generated independently given β ∗ .
Due to the nature of policy optimization, the learning policy π ( a | x ) is expected to be different from β ∗ ( a | x ) , unless β ∗ ( a | x ) is already optimal.
Moreover, in practice the situation could be further complicated.
Again, consider the news recommendation scenario.
Due to the scalability requirement, industrial recommender systems usually adopt a two-stage framework (Ma et al.
(2020)), where one or several candidate generation models first produce a candidate set and a separate ranking model reranks candidate items to present top-K item to users.
While β ∗ ( a | x ) depicts the whole two-stage process, the learning policy π ( a | x ) is usually employed in one particular stage (e.g., the reranking stage), implying drastic differences between the logging and learning policies.
The main challenge of offpolicy learning is then to address the distribution discrepancy between β ∗ ( a | x ) and π ( a | x ) , and learn a policy π ( a | x ) to maximize V ( π ) with access only to the logged dataset D .
One of most widely used methods to address the distribution shift between π ( a | x ) and β ∗ ( a | x ) is the Inverse Propensity Score (IPS)
(Chen et al.
; Munos et al.
One can easily get that:
[Equation]
yielding the following empirical estimator of V ( π ) :
[Equation]
where π ( a n | x n ) /β ∗ ( a n | x n ) is referred to as the propensity score.
In the rest of paper, without further specification, we use the empirical estimation of expectation in our practical calculation.
Various algorithms can be readily used for policy optimization under ˆ V IPS ( π ) , including valuebased methods (Silver et al.
(2016)), policy-based methods (Levine & Koltun (2013); Schulman et al. (2015); Williams (1992)).
In this work, we adopt a well-known policy gradient algorithm,
REINFORCE (Williams (1992)).
Assume the policy π ( a | x ) is parameterized by ϑ , via the “logtrick”, the gradient of ˆ V IPS ( π ϑ ) with respect to ϑ can be readily derived as follows:
[Equation]
Approximation with unknown logging policy .
In many real-world applications, the ground-truth logging policy, i.e., the β ∗ ( a | x ) of each observation ( x , a ) , is unknown.
One reason is the legacy issue, i.e., the probabilities were not logged when collecting data.
Another reason is that the exact value of β ∗ ( a | x ) is intrinsically unavailable such as in the two-stage recommender systems.
As the solution, previous work employs various supervised learning methods (e.g., logistic regression (Schnabel et al.
nerural networks (Chen et al.
(2019), etc.)
to estimate the logging policy, and replaces β ∗ ( a | x ) with its estimated value ˆ β ( a | x ) to get the following estimator for policy learning:
[Equation]
However, as shown in the following proposition, inaccurate ˆ β ( a | x ) leads to high bias and variance of ˆ V BIPS ( π ϑ ) .
Worse still, smaller inaccurate ˆ β ( a | x ) further enlarges this bias and variance.
Proposition 1.
The bias and variance of ˆ V BIPS ( π ϑ ) can be derived as follows:
[Equation]
Smaller ˆ β ( a | x ) usually implies fewer related training samples in the logged data, and thus ˆ β ( a | x ) will be inaccurate with a higher probability.
To make it more explicit, we take KuaiRec dataset (Gao et al.
as an example and estimate the logging policy following (Chen et al.
Figure 1 shows the estimated ˆ β ( a | x ) and its corresponding uncertainties in items of different observation frequencies in the logged dataset.
As uncertainty measures how large the confidence interval is about the current estimation, higher uncertainty implies that the true value may be away from the empirical mean estimate with a high probability.
We defer the discussion about our detailed uncertainty calculation in Section 3.
We can observe from Figure 1 that as item frequency decreases, the estimated logging probability also decreases, but the estimation uncertainty increases.
This implies that smaller ˆ β ( a | x ) is usually 1) more inaccurate and 2) associated with high uncertainty.
As a result, with high bias and variance caused by inaccurate ˆ β ( a | x ) , it is erroneous to improve π ϑ ( a | x ) by simply optimizing ˆ V BIPS ( π ϑ ) .
We propose uncertainty-aware off-policy learning to address this challenge.
U NCERTAINTY - AWARE OFF - POLICY L EARNING
Our idea is incorporating the uncertainty of the logging policy estimation into policy learning.
Observing that
[Equation]
we propose to learn the optimal policy by optimizing the following empirical estimator:
[Equation]
where ϕ x n ,a n is a weight, which reflects ˆ β ( a n | x n ) /β ∗ ( a n | x n ) , to be selected to make ˆ V UIPS ( π ϑ ) as close to V ( π ϑ ) as possible.
Intuitively, one should give small weights to samples whose ˆ β ( a | x ) is far below the ground-truth β ∗ ( a | x ) .
Thus, we divide offline policy improvement into two steps, and repeat them until certain convergence condition is met:
• Uncertainty aware policy evaluation : Derive the optimal uncertainty aware ϕ x ,a to make ˆ V UIPS ( π ϑ ) as accurate as possible.
• Policy Improvement : Learn an improved policy π ϑ ( a | x ) by optimizing ˆ V UIPS ( π ϑ ) .
3. Uncertainty Aware Policy Evaluation
Optimal uncertainty aware weight ϕ x ,a .
We measure the accuracy of ˆ V UIPS ( π ϑ ) by its mean squared error (MSE) to V ( π ϑ ) following previous work (Su et al.
Saito & Joachims (2022)).
MSE captures both the bias and variance of an estimator, since it is the summation of squared bias and variance.
We then locate the ϕ x ,a that can minimize the MSE.
In particular, we demonstrate the optimal ϕ x ,a has a closed-form formula which relates to both the value of π ϑ ( a | x ) /
ˆ β ( a | x ) and the estimation uncertainty of ˆ β ( a | x ) .
More specifically, instead of directly minimizing the MSE, which is intractable, we find the desirable ϕ x ,a by minimizing the upper bound of MSE in the following theorem.
Theorem 1.
Assume r x ,a ∈ [0 , 1] , the mean squared error ( MSE ) between ˆ V UIPS ( π ϑ ) and groundtruth estimator V ( π ϑ ) is upper bounded as follows:
[Equation]
The upper bound in Theorem 1 strictly increases with the two expectations related to ϕ x ,a , which implies that for some choice λ ∈ [0 , ∞ ] , the MSE-optimizing ϕ x ,a can be derived by minimizing:
[Equation]
We cannot directly minimize Eq.(6) since the unknown β ∗ ( a | x ) is involved.
However, various ways (Gal & Ghahramani (2016); Xu et al.
(2021)) can be employed to get the confidence interval which will contain β ∗ ( a | x ) with high probability.
More specifically, following previous work (Joachims et al.
, we assume β ∗ ( a | x ) can be modelled by a softmax function on top of an unknown function f θ ∗ ( x , a ) , i.e., the realizable assumption.
Then we can get:
[Equation]
where f θ ( x , a ) is an estimate of f θ ∗ ( x , a ) .
Following the conventional definition of confidence interval (Abbasi-Yadkori et al. (2011)), we define γ and U x ,a such that | f θ ∗ ( x , a ) − f θ ( x , a ) | ≤ γU x ,a holds with probability at least 1- δ , where γ is a function of δ (typically the smaller δ is, the larger γ is).
Then γU x ,a measures the width of confidence interval of f θ ( x , a ) against its groundtruth f θ ∗ ( x , a ) .
This implies that β ∗ ( a | x ) ∈ B x ,a with probability at least 1- δ , where:
[Equation]
Since β ∗ ( a | x ) can be any value in B x ,a , we adopt the idea of robust optimization (Chen et al.
and find the optimal ϕ x ,a by solving the following optimization problem:
[Equation]
The following theorem derives a closed-form formula for the optimal solution of (8) .
Theorem 2.
Let η 1 , η 2 ∈ [exp( − γU max x ) , exp( γU max x )] , where U max x = max a U x ,a .
The optimization problem in Eq.(8) has a closed-form solution as follows:
[Equation]
Insights on ϕ ∗ x ,a .
The second term of ϕ ∗ x ,a (i.e., 2 η 2 / [exp ( γU x ,a ) + exp ( − γU x ,a )] ) acts like a capping threshold to ensure ϕ ∗ x ,a ≤ 2 η 2 holds even with small π ϑ ( a | x ) /
ˆ β ( a | x ) as shown in Lemma 1 in Appendix A.4.
The key component is the first term, and Lemma 1 implies that:
• If the propensity score π ϑ ( a | x ) / ˆ β ( a | x ) is above the threshold √ λ/η 1 , UIPS will assign a smaller weight to a sample with more inaccurate ˆ β ( a | x ) to prevent its distortion from a large propensity score but an inaccurate logging probability.
• If the propensity score π ϑ ( a | x ) / ˆ β ( a | x ) is below the threshold but not small enough to activate the second term, then the propensity score at the worse case (i
e., taking B − x ,a = ˆ β ( a | x ) ˆ Z exp ( − γU x ,a ) /Z ∗ as denominator) matters.
If the propensity score at the worse case is under control, i.e., π ϑ ( a | x ) / B − x ,a < √ λ , a larger U x,a implies a small propensity score π ϑ ( a | x ) /
ˆ β ( a | x ) , and UIPS tends to boost this safe sample with a higher ϕ ∗ x ,a .
Otherwise ϕ ∗ x ,a still decreases as U x ,a becomes higher.
Uncertainty estimation.
Now we describe how to calculate U x ,a , i.e., the uncertainty of the estimated ˆ β ( a | x ) .
In this work, we choose to estimate β ∗ ( a | x ) using a neural network, due to its encouraging representation learning capacity.
And various ways (Gal & Ghahramani (2016);
Xu et al.
(2021)) can be leveraged to perform the uncertainty estimation in a neural network.
For example, (Gal & Ghahramani, 2016) proposed to estimate uncertainty using dropout; and (Xu et al., 2021) provided a theoretical bound.
Here we adopt the result in (Xu et al.
due to its computational efficiency and theoretical soundness.
Following the proof of Theorem 4.4 in (Xu et al.
(2021)), given the logged dataset D , we can get with high probability ∃ γ :
[Equation]
where g ( x n , a n ) is the gradient of f θ ( x n , a n ) regarding to its last layer, i.e., g ( x n , a n )
= ∇ θ w f θ ( x n , a n ) , where θ w ⊂ θ is the parameter of the last layer of f θ ( x n , a n ) .
And M D = (cid:80)
Nn =1 g ( x n , a n ) g ( x n , a n ) T , implying U x n ,a n = (cid:113) g
( x n , a n ) T M − 1 D g ( x n , a n ) .
3. Policy Improvement
After getting the optimal ϕ ∗ x ,a as in Theorem 2, the policy π ϑ ( a | x ) can be updated by the following
REINFORCE gradient:
[Equation]
[Equation] UIPS then iterates policy evaluation and policy improvement for policy learning until converge.
The whole algorithm framework and important notations are summarized in Algorithm 1 and Table 6 in
Appendix A.1 respectively.
E MPIRICAL E VALUATION
In this section, we evaluate UIPS on both synthetic datasets and three real-world datasets with unbiased data.
We compare UIPS with the following baselines, which can be grouped into five categories:
• Cross-Entropy (CE) :
A supervised learning method with the cross-entropy loss as its objective, which is the commonly used learning approach for a model with softmax output.
No off-policy correction is performed in this method
• IPS-Cap
(Chen et al.
The standard IPS based off-policy learning, which prunes propensity scores to control variance, i.e., taking min( c, π ϑ ( a | x ) ˆ β ( a | x ) ) as the propensity score.
Setting c to a small value can reduce variance, but introduces bias.
• MinVar & stableVar (Zhan et al.
, Shrinkage (Su et al.
This line of work improves off-policy evaluation estimators by reweighing each sample.
For example, MinVar and stableVar reweigh each sample by
[Equation]
since they find that π ϑ ( a | x ) 2 / ˆ β ( a | x ) is directly related to variance.
Su et al.
(2020) proposes to shrink the propensity score by multiplying a weight λ/ ( λ + π ϑ ( a | x ) 2 ˆ β ( a | x ) 2 ) , which is a special case of the proposed UIPS with U x ,a = 0 and η 1 = 1 .
All these work simply treats ˆ β ( a | x ) as β ∗ ( a | x ) , and none of them consider the accuracy or uncertainty of ˆ β ( a | x ) .
• SNIPS (Swaminathan & Joachims (2015c)), BanditNet (Joachims et al.
(2018)), POEM (Swaminathan & Joachims (2015b)), POXM (Lopez et al.
, Adaptive (Liu et al.
This line of work aims for more stable and accurate policy learning.
For example, SNIPS normalizes the estimator by the sum of propensity scores in each batch.
BanditNet extends SNIPS and leverages an additional Lagrangian term to normalize the estimator by an approximated sum of propensity scores of all samples.
POEM jointly optimizes the estimator and its variance.
POXM controls estimation variance by pruning samples with small logging probabilities.
Adaptive proposes a new formulation to utilize negative samples.
• UIPS-P and UIPS-O : Two variants of our proposed UIPS with different ways of leveraging uncertainties.
UIPS-P directly penalizes samples whose estimated logging probabilities are of high uncertainties,
i
e., taking ϕ x ,a = 1 . 0 / exp( γU x ,a ) , which follows previous work on offline reinforcement learning (Wu et al.
; An et al. (2021)).
UIPS-O adversarially uses the worst propensity scores ( π ϑ ( a | x ) /
B − x ,a ) for policy learning, i.e., ϕ x ,a = 1 . 0 / exp( − γU x ,a ) .
4. Synthetic Data
Data generation.
Following previous work (Ma et al.
(2020); Lopez et al.
(2021)), we generate a synthetic dataset by a supervision-to-bandit conversion on Wiki10-31K dataset (Bhatia et al.
(2016)), which is an extreme multi-label classification dataset.
The Wiki10-31K dataset contains approximately 20K samples.
Each sample is associated with a feature vector ˜ x of 101,938 dimensions and a label vector y ˜ x of 31K classes with more than one positive class.
Let y ˜ x ,a denote the label of class a under ˜ x
and we take each class as an action.
We adopt the Wiki10-31K dataset rather than ones in the UCI machine learning repository (Swaminathan & Joachims (2015a)), since it will be much harder with such a large action space.
We then split the dataset into train, validation, test sets with size 11K:3K:6K.
The test set is from the official split.
Since the original feature vector ˜ x is too sparse, for ease of learning, we first embed it to dimension d by x = W ˜ x , and synthesize the ground-truth logging policy β ∗ ( a | x ) by:
[Equation]
where W and { θ ∗ a } are pre-learned parameters by applying a logistic regression model on the train set, τ is a hyper-parameter that controls the skewness of logging distribution.
A small value of τ leads to a near-deterministic policy, while a larger τ makes logging policy smoother.
Due to space limit, more details on data generation and implementation can be found in Appendix A.2.
Evaluation metrics .
To evaluate the learned policy π ϑ ( a | x ) , we calculate Precision@K (P@K),
Recall@K (R@K) and NDCG@K as in previous work (Lopez et al.
; Ma et al.
Higher P@K, R@K and NDCG@K imply a better policy.
Table 1 shows the mean performance and standard deviations of all algorithms under 10 random seeds on three synthetic datasets generated under different τ .
Since the ground-truth logging policy is accessible on the synthetic datasets, we include a new baseline IPS-GT, which depicts the performance the IPS estimator can achieve, assuming the ground-truth logging probabilities are known and sample size is sufficiently large.
We calculate p-value under t-test between UIPS and the best baseline on each dataset to investigate the significance of improvement. First, we can observe that UIPS achieves similar and even better performance than IPS-GT when τ = 0 .
5 and τ = 1 , but
[Table]
[Table]
[Table]
[Table]
performs worse than IPS-GT on the dataset with τ = 2 . Although IPS-GT can access the groundtruth logging probabilities, it still suffers from high variance caused by samples with small logging probabilities, which is the main cause of its worse performance when τ = 0 . 5 and τ = 1 . When the ground-truth logging policy is smoother (e.g., τ = 2 ), the variance of the IPS estimator becomes much smaller
[Table]
and off-policy correction with the ground-truth logging probabilities, rather than the estimated ones, leads to better model performance. We can then observe that as τ increases, i [Table] e., the probability of selecting positive actions decreases, the performance of most algorithms drop, including CE, IPS-Cap, UIPS, Shrinkage, POEM, Adaptive
[Table]
etc. However, UIPS still achieves the best performance on all three datasets under all three metrics.
And as τ decreases, the improvement of UIPS becomes larger and more significant.
SNIPS, BanditNet , POXM are more robust to small logging probabilities of positive actions.
UIPS consistently outperforms Shrinkage (a special case of UIPS with uncertainties always being zero) on all three datasets, demonstrating the benefits of considering the estimation uncertainty.
Finally, regardless of the scale of propensity scores, blindly reweighing through uncertainties also leads to poor performance, as shown by UIPS-P and UIPS-O.
Performance under different uncertainty levels.
As shown in Figure 1, low-frequency actions in the logged dataset suffer higher uncertainties in their propensity estimation.
Thus, we divide the test set into two subsets according to the average frequency of associated actions, where the uncertainty in the subset associated with low-frequency actions is on average 9% higher than that in the subset associated with high-frequency actions.
Table 2 shows the results on these two subsets when τ = 0 . 5 .
We only report the results of the best three baselines due to space limit.
One can clearly observe that only UIPS performed better than CE on the test set associated with low-frequency actions, implying the advantage of UIPS in dealing with the inaccurately estimated logging probabilities. [Table]
Ablation Study.
In this experiment, we aim to answer two questions: (1) Can ˆ V UIPS ( π ϑ ) ) in Eq.
(5) lead to more accurate off-policy evaluation?
(2) How will UIPS perform with different hyperparameters.
Due to space limit, we report results on synthetic dataset with τ = 0 . 5 .
To answer the first question, we evaluate the following ϵ -greedy policy: π ( a | x ) = 1 − ϵ | M x | · I { a ∈
M x } + ϵ/ |A| , where M x contains all positive actions associated with feature vector x .
Then for each x in the test set, we sample 1K data points in a similar way as discussed previously to calculate the value of estimators.
Table 4 shows the MSE of the estimators to ground-truth policy value under 20 different random seeds.
We only compared with baselines on off-policy evaluation estimator, i.e., IPS-Cap, MinVar, StableVar and Shrinkage.
One can observe from Table 4 that UIPS does lead to the smallest MSE, implying the most accurate off-policy evaluation.
For the second question, γ and η 21 /λ are the two most important hyperparameters as discussed in Appendix A.2.1. Thus we fix η 1 , η 2 , and vary λ and γ to track the performance of UIPS.
Recall that a larger γ implies a higher chance the derived interval contains β ∗ ( a | x ) , while √ λ/η 1 is closely related to how UIPS works as discussed in “Insights on ϕ ∗ x ,a ” in Section 3.1.
Figure 3 reports NDCG@5 under diferent γ and λ .
Results on P@5 and R@5 can be found in Appendix A.2.1.
We can observe that to make UIPS perform, B x ,a needs to be of high confidence, e.g., γ =performed the best when τ = 0 .
Moreover, the threshold √ λ/η 1 cannot be too small or too large.
4. Real-World Data
Off-policy learning has its utility in recommendation scenarios (Chen et al.
; Ma et al.
(2020)), where context vector x denotes the state of a user and each candidate item is taken as an action.
To further demonstrate the efficiency of UIPS in real-world scenarios, we evaluate it on three recommendation datasets with unbiased testing data: (1) Yahoo!R3 1 ; (2)Coat 2 ; (3)KuaiRec (Gao et al.
(2022)), from music, fashion and micro-video recommendation scenario respectively.
All these datasets contain an unbiased test set collected from a randomized controlled trial where items are randomly selected.
The statistics of the three datasets and implementation details, e.g., model architectures and dataset splits, can be found in Appendix A.2.2.
We still adopt P@K, R@K and NDCG@K as our evaluation metrics.
Following (Ding et al.
, we take K = 5 on Yahoo!R3 and Coat datasets, and K = 50 on KuaiRec dataset.
The p-value under the t-test between UIPS and the best baseline on each dataset is also reported to investigate the significance of the improvements.
We can first observe that on all three datasets, the proposed UIPS achieves the highest precision, recall and NDCG.
IPS-Cap cannot outperform CE due to the inaccuracy of the estimated logging probabilities.
BanditNet, POEM and POXM tend to perform better with a larger action space, while MinVar, StableVar and Shrinkage as well as Adaptive are more suitable for scenarios with small action size.
UIPS still outperforms Shrinkage, highlighting the importance of modeling uncertainty in the estimated logging policy.
However, reweighing based solely on uncertainties, ignoring the corresponding propensity scores, will also lead to poor performance, as shown by UIPS-P and UIPS-O.
R ELATED W ORK
This work is the first of its kind to take into consideration the uncertainty of the estimated logging policy for improved policy learning.
The following two lines of work are related to this paper.
Off-policy learning.
In many real-world applications, such as search engines, recommender systems, etc., interactive online model update is expensive and risky (Jiang & Li (2016)).
Off-policy learning has therefore attracted increasing interest, since it can leverage the already logged feedback data (Agarwal et al.
; Chen et al.
Liu et al.
The main challenge in off-policy learning is how to address the mismatch between the logging policy and the learning policy.
One line of work (Achiam et al.
(2017); Schulman et al.
(2015)) circumvents this by constraining the learning policy not too far from the logging policy.
However, such constraint is too restrictive thus not applicable in some scenarios such as recommender systems where user behaviors and items change rapidly.
Another more common and widely-applied approach is to leverage Inverse Propensity Score (IPS) method to correct the discrepancy between two policies.
And various methods are proposed for stabilized learning (Swaminathan & Joachims (2015c;a;b)) and variance control (Lopez et al.
(2021); Liu et al.
on top of IPS.
However, all these work directly use the estimated logging policy for off-policy correction, leading to sub-optimal performance as shown in our experiments.
Some other work further extend IPS-based off-policy learning for more complex problems, such as slate recommendation (Swaminathan et al.
two-stage recommender systems (Ma et al.
(2020)), etc.
But they still fail to realize the effect of accuracy of the estimated logging policy.
A recent work (Ding et al.
on causal recommendation also argues that propensity scores may not be correct due to unobserved confounders.
However, they assume the effect of unobserved confounder for any sample can be bounded by a pre-defined hyper-parameter, and adversarially search for the worst-case propensity to update model parameters.
Adapting to off-policy learning, it is a special case of our UIPS-O variant with uncertainty as a pre-defined constant.
Off-policy learning can be directly built on off-policy evaluation.
In this line of research, several work (Su et al.
; Zhan et al.
(2021)) also propose to control the high variance of learning caused by small logging probabilities by instance reweighing.
However, they directly take the estimated logging policy as true logging policy for correction, thus worse than UIPS as shown in experiments.
A recent work (Saito & Joachims (2022)) assumes additional structure in action space and proposes the marginalized IPS.
Instead, our work considers the uncertainty when estimating the logging policy and thus does not add new assumptions about the problem space.
Uncertainty-aware Learning.
Estimation uncertainty has been extensively used for making tradeoffs between exploration and exploitation in online learning (Xu et al.
; Zhou et al.
Abbasi-Yadkori et al.
Recently, several work on offline reinforcement learning (Wu et al.
(2021); An et al.
; Bai et al.
(2022)) penalize the value function of out-of-distribution states and actions by directly subtracting uncertainty to tackle the extrapolating error.
However, blindly penalizing samples of high uncertainty (i.e., UIPS-P) is problematic, as shown in our experiments.
Proper correction depends on both uncertainty in logging policy estimation and the actual value of estimated logging probabilities.
C ONCLUSION
In this paper, we propose a novel Uncertainty-aware Inverse Propensity Score estimator (UIPS) to explicitly model the uncertainty about the estimated logging policy for improved off-policy learning.
UIPS weighs each logged instance to approach the ground-truth estimator and a closed-form solution of the optimal weight is derived by minimizing the upper bound of the mean squared error (MSE).
An improved policy can be obtained by optimizing the resulting estimator.
Extensive experiments on synthetic datasets and three real-world datasets demonstrate the efficiency of UIPS .
As demonstrated in this work, explicitly modeling the uncertainty of the estimated logging policy is crucial for effective off-policy learning; but the best use of this uncertainty is not to simply downweigh or drop instances with uncertain estimations, but to balance it with the actually estimated logging probabilities in a per-instance basis.
As our future work, it is promising to investigate
how
UIPS can be extended to value-based learning methods, e.g., actor-critics.
And on the other hand, it is also important to analyze how tight our upper bound analysis of MSE is; and if possible, find new ways to tighten it for improvements.
A A PPENDIX A. N OTATIONS AND A LGORITHM F RAMEWORK .
For ease of reading [Table]
we list important notations in Table 6 and summarize the main framework of the proposed UIPS in Algorithm
[Table]
V (
[Table]
[Table]
D := { ( x n , a n , r x n ,a n ) | n ∈ [ N ] }
[Table]
N
[Table]
ϕ ∗ x ,a
[Table]
f θ ∗ ( x , a ) the unknown ground-truth function that generates β ∗ ( a | x ) =
exp( f θ ∗ ( x ,a )) (cid:80) a ′ exp( f θ ∗ ( x ,a ′ ))
f θ ( x , a ) the estimate of f θ ∗ ( x , a ) that generates ˆ β ( a | x )
B x ,a confidence interval of ˆ β ( a | x )
U x ,a uncertainty defined as | f θ ∗ ( x , a ) − f θ ( x , a ) | ≤ γU x ,a g ( x n , a n ) gradient of f θ ( x , a ) regarding to the last layer.
Computation Cost.
The additional computation cost of UIPS over IPS comes from two parts:
• Pre-calculating uncertainties (line 1-5 in Algorithm 1) :
This part calculates uncertainty of the logging probability for each ( s, a ) pair, and “it only needs to be executed once”.
The computational cost of this step is O
( Nd 2 + d 3 ) , where O ( Nd 2 ) is for calculating uncertainties in each ( s, a ) pair and O ( d 3 ) is for matrix inverse.
• Calculating ϕ ∗ x ,a during training (line 8 in Algorithm 1): It only takes O(1) time, the same computational cost as calculating IPS score.
Note that calculating logging probability for each sample, which is essential for both UIPS and IPS, takes O ( Nd |A| ) time.
Since the dimension d is usually much less than action size |A| and samples size N , UIPS does not introduce significant computational overhead compared to the original IPS solution.
A. E XPERIMENTS D ETAILS A.2. S YNTHETIC D ATA
Data generation.
Given the ground-truth logging policy β ∗ ( a | x ) , we generate the logged dataset as follows.
For each sample in train set, we first get the embedded context vector x form its original
Main part of
feature vector ˜ x .
We then sample an action a according to β ∗ ( a | x ) , and obtain the reward r x ,a = y ˜ x ,a , resulting a bandit feedback ( x , a, r x ,a ) , where y ˜ x ,a is the label of class a under the original feature vector ˜ x .
We repeat above process N times to collect the logged dataset.
In our experiments, we take d = 64 , N = 100 .
Implementation Details.
We model the logging policy as in Equation (7) with f θ ( x , a ) =
x T θ a , where { θ a } are parameters to learn.
To train the logging policy, we take all samples in the logged dataset D as positive instances, and randomly sample non-selected actions as negative instances as in (Chen et al.
We use grid search to select the hyperparameters based on the model’s performance on validation dataset: the learning rate was searched in { 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 } ; λ, γ, η 1 were searched in { 0.5, 0.1, 1, 2,5, 10, 15, 20, 25, 30, 40, 50 } .
And η 2 was searched in { 1, 10, 100, 1000 } .
For baseline algorithms, we perform a similar grid search as mentioned above, and the search range follows the original papers.
Ablation Study: Hyperparameter tuning.
Although UIPS has four hyperparameters ( λ , γ , η 1 , and η 2 ), one only needs to carefully finetune two of them, i.e., γ and η 21 /λ , to obtain good performance of UIPS.
This is because:
• η 2 acts like a capping threshold to ensure ϕ ∗ x ,a ≤ 2 η 2 holds even with small propensity scores.
Hence, it should be set to a large value (e.g., 100).
• The key component (i
e., the first term) of ϕ ∗ x ,a can be rewritten in the following way.
While all ( x , a ) pairs will be multiplied by ϕ ∗ x ,a , η 1 in the numerator will not affect final performance too much, and the key is to find a good value of η 21 /λ to balance the two terms in the denominator:
[Equation]
Thus with η 1 and η 2 fixed, effect of hyperparameter γ and λ on precision and recall can be found in
Figure 2a and Figure 2b respectively.
A.2. R EAL - WORLD D ATA
Statistics of data.
The statistics of three real-world recommendation datasets with unbiased data can be found in Table 7.
All these datasets contain a set of biased data collected from users’ interactions on the platform, and a set of unbiased data collected from a randomized controlled trial where items are randomly selected.
As in (Ding et al.
(2022)), on each dataset, the biased data is used for training, and the
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
unbiased data is for testing, with a small part of unbiased data split for validation purpose (5% on
Yahoo and Coat, and 15% on KuaiRec).
We take the reward as 1 if : (1) the rating is larger than 3 in Yahoo!R3 and Coat datasets; (2) the user watched more than 70% of the video in KuaiRec.
Otherwise, the reward is labeled as 0.
Implementation Details.
We adopt a two-tower neural network architecture to implement both the logging and learning policy, as shown in Figure 3.
For the learning policy, the user representation and the item representation are first modelled through two separate neural networks (i.e., the user tower and the item tower), and then their element-by-element product vector is projected to predict the user’s preference for the item.
We then re-use the user state generated from the user tower of the learning policy, and model the logging policy with another separate item tower, following (Chen et al.
We also block gradients to prevent the logging policy interfering the user state of the learning policy.
In each learning epoch, we will first estimate the logging policy, and then take the estimated logging probabilities as well as their uncertainties to optimize the learning policy.
All hyperparameters are searched in a similar way described in Section 4.1.
A. E XPERIMENTS ON THE D OUBLY R OBUST E STIMATORS .
The doubly robust (DR) estimator (Jiang & Li (2016)), which is a hybrid of direct method (DM) estimator and inverse propensity score (IPS) estimator, is also widely used for off-policy evaluation.
More specifically, let ˆ η : X ×
A → R be the imputation model in DM that estimates the reward of action a
under context vector x , and ˆ β ( a | x ) be the estimated logging policy in the IPS estimator.
The
DR estimator evaluates the policy π based on the logged dataset D := { ( x n , a n , r x n ,a n ) | n ∈ [ N ] } , by:
[Equation]
where ˆ V DM ( π ) is the DM estimator:
[Equation]
[Equation]
Again assuming the policy π ( a | x ) is parameterized by ϑ , the REINFORCE gradient of ˆ V DR ( π ϑ ) with respect to ϑ can be readily derived as follows:
[Equation]
The imputation model ˆ η ( x , a ) is pre-trained following previous work (Liu et al.
with the same model architecture as the logging policy model.
Besides the standard DR estimator, we also adapt UIPS and the best two baselines on off-policy evalaution estimator (i.e., MinVar and Shrinkage) to doubly robust setting using the same imputation model.
Table 8 and Table 9 show the results on the synthetic datasets and three real-world datasets respectively.
For ease of comparison, we also include the experimental results of IPS-Cap and UIPS on each dataset in two tables.
Two p-values are also provided: (1) P-value(UIPSDR): The p-value under the t-test between UIPSDR and the best DR baseline on each dataset; (2) P-value(UIPS): The p-value under the t-test between UIPS and the best DR baseline on each dataset.
From Tableand Table 9, we can first observe that DR cannot consistently outperform IPS-Cap: It outperforms
IPS-Cap on the Coat and KuaiRec dataset, while achieving much worse performance on the synthetic datasets and Yahoo dataset.
This is because the imputation model also plays an important role in gradient calculation as shown in Equation(13), so its accuracy greatly affects policy learning. When the imputation model is sufficiently accurate, for example, on the Coat dataset with only 300 actions, incorporating the DM estimator not only leads to better performance of DR over IPS, but also improved performance of UIPSDR over UIPS. And in particular, in this situation UIPSDR performs better than DR with the gain being statistically significant. When the imputation model is not accurate enough, for example, on the KuaiRec dataset with a large action space but sparse reward feedback, DR is still worse than UIPS, and UIPSDR also performs worse than UIPS due to the distortion of the imputation model.
A. T HEORETICAL P ROOF .
Proof of Proposition 1:
Proof.
With the linearity of expectation, we have E D (cid:104) ˆ V BIPS ( π ϑ ) (cid:105) = E β ∗
[Equation]
π ϑ ( a | x ) ˆ β ( a | x ) r x ,a (cid:105) , thus: [Equation] [Equation] [Table]
[Figure]
[Table] [Table]
For variance, since samples are independently sampled from logging policy, thus :
[Equation]
By re-scaling
[Equation]
we get:
[Equation]
Then we complete the proof.
Proof of Theorem 1:
Proof.
We can get:
[Equation]
ˆ V UIPS ( π ϑ ) − V ( π ϑ ) (cid:17) 2 (cid:21) = (cid:16) E D (cid:104)
ˆ V UIPS ( π ϑ ) − V
{+( π ϑ ) (cid:105)(cid:17) 
+}
+ Var D (cid:16)
ˆ V UIPS ( π ϑ ) − V ( π ϑ ) (cid:17) =
(cid:16) E D (cid:104)
ˆ V UIPS ( π ϑ ) − V
{+( π ϑ ) (cid:105)(cid:17) 
+}
+ Var D (cid:16)
ˆ V UIPS ( π ϑ ) (cid:17) =
{+Bias( ˆ V UIPS ( π ϑ )) 
+}
+ Var( ˆ V UIPS ( π ϑ )) .
We first bound the bias term:
[Equation]
Equality (1) follows the linearity of expectation.
Inequality (2) is due to the Cauchy-Schwarz inequality.
We then bound the variance term:
[Equation]
[Equation]
Combining the bound of bias and variance, we can complete the proof.
Proof of Theorem 2:
Proof.
We first define several notations:
• T ( ϕ x ,a , β x ,a )
[Equation]
λ E β
[Equation]
• ˜ T ( ϕ x ,a ) =
max β x ,a ∈ B x ,a T ( ϕ x ,a , β x ,a ) denotes the maximum value of inner problem.
• T ∗ =
min ϕ x ,a ˜ T ( ϕ x ,a ) =
min ϕ x ,a max β x ,a ∈ B x ,a T ( ϕ x ,a , β x ,a ) denote the optimal minmax value.
And ϕ ∗ x ,a = arg min ϕ x ,a ˜ T ( ϕ x ,a ) .
• B − x ,a :=
ˆ Z exp( − γU x ,a ) Z ∗ ˆ β ( a | x ) , and B + x ,a :=
ˆ Z exp( γU x ,a ) Z ∗ ˆ β ( a | x ) .
We first find the maximum value of inner problem, i.e., ˜ T ( ϕ x ,a ) for any fixed ϕ x ,a .
And there are three cases shown in Figure 4:
Case I: When ˆ β ( a | x ) ϕ x ,a ≥ B + x ,a , , ˜ T ( ϕ x ,a ) achieves the maximum value at β x ,a = B − x ,a .
In other words, ˜ T (
Case II: When B − x ,a ≤ ˆ β (
[Equation]
, then ˜ T ( ϕ x ,a ) will be the maximum between T ( ϕ x ,a , B − x ,a ) and T ( ϕ x ,a , B + x ,a ) .
More specifically, when ˆ β ( a | x ) ϕ x ,a
[Equation]
−
[Equation]
i.e.
[Equation]
Otherwise when ϕ x ,a <
[Equation]
Case III: When ϕ x ,a ≥ ˆ β ( a | x ) B − x
[Equation]
implying
[Equation]
a
[Equation]
x
[Equation]
Overall, we get that:
[Equation]
Next we try to find the minimum value of ˜ T ( ϕ x ,a ) .
We first observe that without considering constraint on ϕ x ,a , when
[Equation]
T ( ϕ x ,a , B + x ,a ) achieves the global minimum value.
However, ϕ + x ,a ≤ ˆ β ( a | x ) B + x ,a , which implies when ϕ x ,a ∈ [ 2ˆ β ( a | x ) B + x ,a + B − x ,a , ∞ ) , the minimum value of T ( ϕ x ,a , B + x ,a ) achieves at 2ˆ β ( a | x ) B + x ,a + B − x ,a .
On the other hand, without considering any constraint on ϕ x ,a , global minimum value of
T ( ϕ x ,a , B − x ,a ) achieves at:
[Equation]
Thus if ϕ − x ,a ≤ 2ˆ
[Equation]
Otherwise ϕ ∗ x ,a = 2ˆ
[Equation]
Overall,
[Equation]
  ( [Equation] We use η 1 and η 2 to represent Z ∗ ˆ Z in the two terms respectively.
We can get ϕ
∗ x
[Equation]
(cid:32) [Equation] ( − γU x ,a ) (cid:33) (21) where η 1 , η∈ [exp( − γU max s ) , exp( γU max s )] , since ˆ Z exp( − γU max s ) ≤ Z ∗ =
[Equation]
a [Equation] exp( f
[Equation]
( a ′ | x )) ≤ ˆ Z exp( U max s ) .
Usually we set η 1 ≤ η 2 .
We introduce two parameters since the scale of η 1 is closely related to the scale of λ , while the scale of η 2 is independent.
Then we complete the proof .
Lemma 1.
Assume
η≤ η 2 , then with fixed π ϑ ( a | x ) and ˆ β ( a | x ) , and α x ,a = (cid:113) λ
[Equation]
exp( − 2 γU x ,a ) , we have the following observations:
• If π ϑ ( a | x ) ˆ β ( a | x ) ≤ α x ,a , ϕ ∗ x ,a = 2 η 2 / [exp( γU x ,a ) + exp( − γU x ,a )] .
Otherwise ϕ ∗ x ,a = λ/ (cid:104) λη 1 exp ( − γU x ,a ) + η 1 π
ϑ
( a | x ) 2 ˆ β 2 ( a | x )exp( − γU x ,a ) ] (cid:105) .
In other words, ϕ ∗ x ,a ≤ 2 η 2 always holds.
• If π ϑ ( a | x ) ˆ β ( a | x ) ≥ √ λη 1 , then ϕ ∗ s,a will always decrease as U x ,a increases.
• If α x ,a ≤ π ϑ ( a | x ) ˆ β ( a | x ) < √ λη 1 exp( − γU x ,a ) , larger U x ,a brings larger ϕ ∗ x ,a .
Otherwise ϕ ∗ x ,a still decreases as U x ,a increases.
Proof.
For the first observation, deriving
[Equation]
we can get the result.
For the second and third observation, since η 1 ≤ η 2 , then α x ,a ≤ √ λ √ 2 η 1 ≤ √ λη 1 .
Let L ( u ) = λη 1 exp [Equation] − γu [Equation]
+
[Equation]
[Equation]
a
[Equation]
a [Equation] x [Equation]
exp
[Equation]
) , we can have:
[Equation]
By letting ∇ u L ( u ) ≥ 0 , we can get u ≥ 1 γ log (cid:16) √ λ ˆ β ( a | x ) η 1 π ϑ ( a | x ) (cid:17) .
This implies when U x ,a ≥ 1 γ log (cid:16) √ λ ˆ β ( a | x ) η 1 π ϑ ( a | x ) (cid:17) , ϕ ∗ x ,a will decrease as U x ,a increases.
Otherwise as U x ,a increases, ϕ ∗ x ,a also increases.
Specially, when π ϑ ( a | x
[Equation]
≤ 0 .
Given U x ,a ≥ 0 , this implies that ϕ ∗ x ,a will always decrease as U x ,a increases in this case.
Otherwise, when α x ,a ≤ π ϑ ( a | x ) ˆ β ( a | x ) ≤ √ λη 1 exp( − γU x ,a ) , with [Equation] ( a | x
[Equation]
larger U x ,a implies larger ϕ x ,a .
Otherwise ϕ ∗ x ,a still decreases as U x ,a increases.
This completes the proof.