Generalizing Off-Policy Learning under Sample Selection Bias
Abstract
Learning personalized decision policies that generalize to the target population is of great relevance.
Since training data is often not representative of the target population, standard policy learning methods may yield policies that do not generalize target population.
To address this challenge, we propose a novel framework for learning policies that generalize to the target population.
For this, we characterize the difference between the training data and the target population as a sample selection bias using a selection variable.
Over an uncertainty set around this selection variable, we optimize the minimax value of a policy to achieve the best worst-case policy value on the target population.
In order to solve the minimax problem, we derive an efficient algorithm based on a convex-concave procedure and prove convergence for parametrized spaces of policies such as logistic policies.
We prove that, if the uncertainty set is well-specified, our policies generalize to the target population as they can not do worse than on the training data.
Using simulated data and a clinical trial, we demonstrate that, compared to standard policy learning methods, our framework improves the generalizability of policies substantially.
INTRODUCTION
Learning personalized policies has become integral to modern decision-making in a variety of domains such as medicine [Hill and Su, 2013] and public policy [Kube et al., 2019].
Since in these domains exploration is costly or otherwise infeasible, many methods have been proposed for off-policy learning, i e., policy learning from existing data [e. g., Dudík et al., 2014, Kallus, 2018, Athey and Wager, 2021, Tschernutter et al., 2022]
A major challenge in off-policy learning is the generalizability of policies.
Generalizability is concerned with whether a policy learned on the data for training (i e., training data) is also effective in the target population.
Standard methods for policy learning yield policies that are effective on the target population, if, and only if, the training data is representative of the target population [e. g., Beygelzimer and Langford, 2009, Dudík et al., 2014].
However, this may not hold true in practice [e. g., Buchanan et al., 2018, Cole and Stuart, 2010,
Downs and Black, 1998, Flores et al., 2021, Norris et al., 2001, Rothwell, 2005].
For instance, a review of HIV/AIDS clinical trials found that women are largely underrepresented in these trials [Gandhi et al., 2005, Greenblatt, 2011], so that data from these trials is not representative of the actual target population (i.e., the population of HIV-positive patients in the USA).
Therefore, when data from such trials is used to derive policies, standard methods for policy learning may not generalize to the target population.
As such, these policies may be ineffective or even harmful on the target population and, therefore, not relevant in practice.
In this paper, we develop a framework for learning policies from training data that generalize to the target population.
For this, we characterize the difference between training data and target population as a sample selection bias using an unknown selection variable [e. g., Cortes et al., 2008,
Manski, 1989].
If we had oracle access to the true selection variable, we could re-weight the data accordingly in order to obtain the value of a policy on the target population.
Since, in practice, the true selection variable is unknown, the value of a policy on the target population is not identifiable from training data.
Instead, we derive bounds on the odds-ratio of the selection probability, which yields an uncertainty set around the true selection probabilities.
Then, our framework optimizes the minimax value of a policy to achieve the best worst-case policy value on the target population.
We prove that, if the uncertainty set is well-specified, our framework yields policies that do not do worse on the target population than the worst-case policy value estimated from the training data.
As such, these policies can generalize to the target population.
In order to efficiently optimize the minimax value of a policy, we show that it can be written as a difference of convex functions (DC) program.
Then, by leveraging the structure of the adversarial subproblem, we develop a tailored m ini m ax c onvex- c oncave p rocedure (MMCCP).
We prove that MMCCP converges for certain parameterized spaces of policies such as logistic policies.
Using synthetic data and a clinical trial, we demonstrate that standard policy learning methods generalize poorly, while our framework improves the generalizability of policies substantially.
As such, our framework enables to learn reliable policies that can be implemented in the target population.
PRELIMINARIES
In this section, we describe the setup, formulate the problem of generalizing policies, and discuss related work.
2.1 SETUP
We consider a binary treatment T P t 0 , 1 u , covariates
X P X (cid:132) R d , and the outcome Y P R .
We use the convention that lower outcomes are preferred.
Using the NeymanRubin potential outcomes framework [Rubin, 2005], let
Y p 0 q , Y p 1 q be the potential outcomes for each of the treatments.
Further, let a policy π be a map from the covariates to the probability of treatment assignment, i. e., π : X (cid:209) r 0 , 1 s .
Then, the policy value of π is given by
[Equation]
The objective of policy learning is to find a policy π in a policy class Π that minimizes the policy value, i. e., π P arg min π P Π V p π q .
We make the following three standard assumptions [Rubin, 1974]: (i) consistency (i. e., Y (cid:16) Y p T q ); (ii) positivity (i. e., [-
-]0 (cid:160) p p T (cid:16) 1 | X (cid:16) x
q (cid:160) 1 for all x ); and (iii) strong ignorability (i. e., Y p {+
+} q , Y p 1 q KK T | X ).
Then we can identify the policy value in (1) in terms of the observed data p X, T, Y q .
2.2 PROBLEM FORMULATION
Suppose we are interested in learning a policy that minimizes the policy value under the target distribution p X, T, Y q (cid:18) P , i. e., V Target p π q .
However, we are not given data from the target distribution, but only data from a (potentially different) training distribution p X, T, Y q (cid:18) P Train .
Standard policy learning methods assume that the training and target distributions are identical.
However, even in carefully designed clinical trials, the subjects in the trial are often not representative of the target population, i. e., P Train (cid:24) P [e. g., Buchanan et al., 2018, Cortes et al., 2008, Downs and
Black, 1998, Flores et al., 2021, Gandhi et al., 2005, Greenblatt, 2011, Rothwell, 2005].
Hence, standard methods for policy learning yield policies that minimize the policy value on the training data, i. e., V Train p
π q (cid:16) E Train
r Y π s .
However, since the policy value depends on the underlying data distribution, these policies may not minimize the policy value on the target population, i. e., V Target p
π q (cid:16) E r Y π s .
This can be seen when writing E r Y π s in terms of the distribution P Train using a change of probability measure:
[Equation]
where the random variable R (cid:16) d P { d P Train is the RadonNikodým derivative , 2 also know as density ratio .
As a direct implication, if P Train (cid:24) P and, thus, R (cid:24) 1 , it follows that
[Equation]
In other words, a policy learned from training data using standard methods may not generalize to the target population, and, as such, may be of little help in practice.
In this paper, we consider the realistic setting in which the training data is not representative of the target population.
We propose a framework for learning policies that generalize to the target population only given data from the training distribution, i. e., tp [Equation] [Equation]
2.3 RELATED WORK
Despite the vast literature on off-policy learning, less work considers the problem of learning policies that generalize to the target population.
Below, we summarize works on offpolicy learning and works on external validity in causality, which is closely related to generalizability.
Off-policy learning.
Off-policy learning methods can be broadly divided into three categories:
(i) Direct methods estimate the outcome functions
[Equation]
t
[Equation] | [Equation] x s and plug them into (1), i. e., ˆ
[Equation]
DM
[Equation]
1 n
[Equation]
[e. g., Bennett and Kallus, 2020].
This approach is closely related to estimating the conditional average treatment effect, i. e., E r Y p 1 q (cid:1)
{+Y p 
+}
q | X s [Shalit et al., 2017, Hatt et al., 2022].
Direct methods are known to be weak against model misspecification with regards to µ t p x q .
(ii)
Weighting methods re-weight the outcome data such that it looks as if it were generated by the policy that is evaluated [e. g., Bottou et al., 2013, Horvitz and Thompson, 1952, Kallus, 2018, Li et al., 2011].
A common choice for weights are the normalized inverse propensity weights
[Swaminathan and
Joachims, 2015], i.
[Equation]
[Equation]
[Equation]
q , where
W
[Equation]
1 {pp 1 (cid:1) 2 T
[Equation]
qp 1 (cid:1) [Equation] (cid:1)
[Equation]
p X i qqq and π
[Equation]
p x q (cid:16) P p T (cid:16) 1 | X (cid:16) x q is the so-called behav- ior policy, which was used to generate the training data.
(iii) Doubly robust methods combine direct and weighting methods typically using the augmented inverse propensity weight estimator
[Athey and Wager, 2021,
Dudík et al., 2014, Thomas and Brunskill, 2016].
When the direct estimate of ˆ µ t is biased, the doubly robust method weights the residuals by the inverse propensity weights in order to remove the bias, i. e., ˆ V [Equation] p [Equation]
q
[Equation]
The above methods have become the standard for off-policy learning.
Despite their widespread use, the above methods implicitly assume that the training data, which is used to learn the policy, is representative of the target population.
As such, when the training data is not representative of the target population, we cannot rely on the above methods, as policies may not generalize to the target population.
Distributionally robust optimization.
A related, yet fundamentally different idea is distributionally robust optimization (DRO) [e. g., Duchi and Namkoong, 2018], which studies robustness towards distributional shifts.
DRO has found application in off-policy learning by optimizing worst-case policy values [Si et al., 2020] and individualized treatment rules [Zhao et al., 2019b, Mo et al., 2020].
While generalizability is related to DRO, since the difference between training and target distribution can be seen as a distributional shift, it is fundamentally different, as DRO allows for arbitrary changes in distribution.
In contrast, generalizability considers a training distribution that is, potentially, not representative of the target population, but derived from the target population.
That is, generalizability considers differences in the distributions the arise from an unknown selection mechanism into the training data.
Moreover, DRO and its applications require the decision-maker to quantify the distance between training and target distribution in terms of some divergence measure (typically the Kullback-Leibler divergence), which may be notoriously difficult for domain experts such as clinicians.
In contrast, our approach allows for user-friendly and intuitive calibration of the parameters involved in the uncertainty set due to recognizing that the differences arises from an unknown selection mechanism.
External validity in causality.
Different to policy learning, causal inference aims to estimate causal effects from observational data [Bottou et al., 2013, Kuzmanovic et al., 2021,
Hatt and Feuerriegel, 2021].
External validity in causal inference is concerned with whether causal effect estimates obtained from a study sample are also valid for the target population.
A common approach to address this is to reweight the data with the inverse of a subject’s probability to be selected into the study sample [e. g., Buchanan et al., 2018, Cole and Stuart, 2010, Dahabreh et al., 2019, Imai et al., 2013, Stuart et al., 2011].
This idea has been extended to a doubly robust method for off-policy learning [Uehara et al., 2020].
Predominantly used in economics, the Heckman correction is another technique that is also based on a subject’s selection probability [Heckman, 1979].
However, in order to estimate these selection probabilities, all existing approaches assume that data from both the study sample and the target population is given.
In practice, however, we are only given data from the study sample and not from the target population.
Other approaches include approximations of the bias arising from the difference in the study sample and target population by using weights that do not depend on the selection variable [Andrews and Oster, 2017], by bounding the weights directly
[Aronow and Lee, 2013], or, in addition, by constraining the shape of the population outcome distribution [Miratrix et al., 2018].
Different to the above approaches and more practically, we do not assume that we have access to samples from the target population and, therewith, we cannot estimate a subject’s selection probability.
As a remedy, we present our framework for learning generalizable policies in the following.
GENERALIZING OFF-POLICY LEARNING UNDER SAMPLE SELECTION BIAS
In this section, we introduce our framework for learning policies that generalize to the target population.
For this, we first characterize the difference between the training distribution P Train and the target distribution P as a sample selection bias (Section 3.1).
Then, based on this, we derive an uncertainty set and optimize the minimax policy value to achieve the best worst-case policy value (Section 3.2).
We prove that policies learned in this way do not do worse on the target population than the worst-case policy value and, as such, can be generalized to the target population (Section 3.3).
3.1 SAMPLE SELECTION BIAS
In this section, we characterize the difference between the training distribution P Train and the target distribution P as a sample selection bias using a selection variable [e. g., Cortes et al., 2008, Manski, 1989].
This then allows us to characterize the Radon-Nikodým derivative R (cid:16) d P { d P Train in (2) in terms of the selection variable.
We represent the selection bias with a selection variable
S P t 0 , 1 u .
If, for a subject, S (cid:16) 1 , the subject is included in the training data, and, if S (cid:16) 0 , the subject is excluded from the training data.
As a result, we can write the training distribution in terms of the target distribution:
[Equation]
Based on this, we characterize the Radon-Nikodým deriva- tive, which enables us to write the policy value on the target population in terms of the selection variable S and the training distribution P Train .
Proposition 1.
Under the sample selection bias, we can write the Radon-Nikodým derivative R (cid:16) d P { d P Train as
[Equation]
and, therefore, we can write the policy value on the target population as
[Equation]
See Appendix A.1 for a proof.
If, hypothetically, we observed S , we could estimate R (cid:16) P p S (cid:16) 1 q { P p S (cid:16) 1 | X, T, Y q 3 and re-weight the data accordingly in order to obtain the policy value on the target population.
However, we never observe the selection variable S , since we only observe the training data for which S (cid:16) 1 .
This renders the selection variable S unidentifiable from the training data.
Instead, we use an uncertainty set over which we optimize the minimax policy value on the target population.
3.2 LEARNING GENERALIZABLE POLICIES BY OPTIMIZING MINIMAX POLICY VALUE
We derive an uncertainty set around R (cid:16) P p S (cid:16) 1 q { P p S (cid:16)| X, T, Y q over which we maximize the policy value to obtain the worst-case policy value.
Then, our framework optimizes the minimax policy value to achieve the best worst-case policy value on the target population.
If we had oracle access to the true Radon-Nikodým derivative R (cid:6) i (cid:16) R (cid:6) i p X i , T i , Y i q , we could estimate the policy value on the target population using Proposition 1, that is, by re-weighting the data with R (cid:6) .
This often leads to high variance estimates due to probabilities close to zero.
As a remedy, since E r R (cid:6) s (cid:16) 1 , we use the empirical sum of the true Radon-Nikodým derivatives as a control variate to normalize the estimate.
This gives rise to the following
Hajek estimator for the policy value on the target population
V Target p π q :
[Equation]
where ψ i p π q corresponds to one of the three standard methods for policy learning: direct, weighting, and doubly robust methods.
Formally, ψ i p π q is either ψ DM i
[Equation]
or ψ
[Equation]
p π q given as:
[Equation]
The outcome functions µ t p x q and the weights W IPW need to be estimated from data.
Any ψ p π q in (8), (9), or (10) can be chosen for estimating the policy value as long as the data is re-weighted with the Radon-Nikodým derivative R (cid:6) .
Since the true R (cid:6) is unknown, we instead derive a worstcase policy value on the target population.
This allows to ensure that our policy does not do worse than expected once it is implemented in the target population.
For this, we maximize (7) over an uncertainty set around R (cid:6) .
We consider an uncertainty set motivated by sensitivity analysis in causality [e. g., Kallus et al., 2019, Kallus and Zhou, 2018,
Rosenbaum, 2002, Zhao et al., 2019a], which restricts by how much P p S (cid:16) 1 | X, T, Y q can vary from P p S (cid:16) 1 q via the odds-ratio characterization as follows:
[Equation]
where Γ ¥ 1 .
For Γ (cid:16) 1 , we have equal probability of selection, i. e., P p S (cid:16) 1 | X, T, Y q (cid:16) P p S (cid:16) 1 q and, thus, no difference between the training data and the target population.
Larger values of Γ allow for larger variation in the probabilities of selection.
The bounded odds-ratio in (11) immediately yields an uncertainty set for the RadonNikodým derivative:
[Equation]
where l
[Equation]
where u (cid:16) Γ [Equation] P p [Equation]
[Equation]
The uncertainty set R includes all Radon-Nikodým derivatives R that satisfy the odds-ratio restriction in (11).
For a given policy, we seek the maximum policy value on the target population among all Radon-Nikodým derivatives in the uncertainty set.
This yields the following worst-case policy.
Definition 1.
(Worst-case policy value.)
The worst-case policy value on the target population under the bounded odds-ratio with parameter Γ is given by
[Equation]
where ψ i p π q corresponds to either (8) , (9) , or (10) .
Then, we seek the optimal policy in a policy class Π , which minimizes the worst-case policy value on the target population, i. e.,
[Equation]
In particular, a policy learned with our framework generalizes to the target population, since it does not do worse on the target population than the worst-case policy value estimated using the training data.
For this, a decision-maker only has to quantify the population selection probability, i. e., P p S (cid:16) 1 q and appropriately choose the maximum deviation from it via Γ .
We discuss data-driven approaches to choose these quantities in Section 3.4.
We derive a tailored convex-concave procedure for optimizing (16) in Section 4.
3.3 THEORETICAL GUARANTEES FOR GENERALIZABILITY
We prove that, if the Radon-Nikodým is appropriately bounded, the worst-case policy value, V Target p π ; R q , is asymptotically an upper bound for the true policy value on the target population, V Target p π q .
As such, a policy learned with our framework does not do worse on the target population than the worst-case policy value.
Similar to [Athey and Wager, 2021], we express the flexibility of a policy class Π using the notion of the Rademacher complexity, i. e.,
R n p Π q .
Theorem 1.
(Generalization bound.)
Suppose the true
Radon-Nikodým derivative is appropriately bounded, i. e.,
R (cid:6) P R and, therefore, l ⁄ R (cid:6) i ⁄ u , and we have bounded outcomes, i. e., | Y | (cid:160) C .
Then, for a constant K ψ depending on ψ p π q and for some δ ¡ 0 , we have that,
[Equation]
(17) with probability at least 1 (cid:1) δ and for any π P Π .
See Appendix A.2 for a proof.
All policy classes we consider have ?
n -vanishing Rademacher complexity, i. e.,
R n p Π q (cid:16) O p n (cid:1) 1 { 2 q .
Therefore, Theorem 1 proves that, asymptotically, V Target p
π q is an upper bound for V Target p
π q .
This guarantees that π p Π , R q from (16) does not do worse on the target population than the worst-case policy value, which is calculated using training data.
In particular, since π p Π , R q minimizes the right hand side of (17), π p Π , R q is the best policy that guarantees to generalize to the target population.
Our bound in Theorem 1 holds without complete knowledge of the selection variable and proves that our framework yields policies that generalizes to the target population.
Note that in Theorem 1, we use the true nuisance functions instead of estimates, since it has been shown that this does not affect the leading term in the convergence rate of the policy value (see Athey and Wager [2021]; Sec. 3.1, Sec. 3.2, and Lemma 4).
This holds true if the nuisance functions have finite second moment
and we use consistent estimators for the nuisance functions and L 2 errors decay with 1 { n ζ , where ζ depends on the nuisance functions.
Hence, to provide a generalization bound on the policy value, it is enough to consider the true nuisance functions as we did in Theorem 1.
3.4 CALIBRATION OF Γ AND P p S (cid:16) 1 q
In this section, we discuss two approaches to calibrate the parameters Γ and P p S (cid:16) 1 q in (11), which are context dependent: (i) Practitioner calibration with domain knowledge and (ii data-driven calibration.
(i) Practitioner calibration: This approach is based on domain knowledge of practitioners about variables that impact selection into training data.
First, P p S (cid:16) 1 q , the population probability of inclusion, needs to be quantified.
If the study is randomized, a value (cid:19) 1 { 2 is reasonable.
Second, Γ , the largest deviation from P p S (cid:16) 1 q , needs to be quantified.
Our framework allows a practitioner-friendly choice of calibration parameters.
In fact, both questions may be simply answered using domain knowledge.
(ii) Data-driven calibration: Although our framework enables practitioners to choose appropriate calibration parameters, we provide a fully data-driven approach for calibrating Γ and P p S (cid:16) 1 q .
To this end, we consider a setting in which samples from one of the covariates of the target population are provided.
This is reasonable, since we often have limited understanding of the target population and, for instance, know covariates such as the distribution of gender or age in the target population.
Once we are given one covariate, e. g., x age , we proceed in two steps: (1) For calibrating P p S (cid:16) 1 q , we approximate P p S (cid:16) 1 | X, Y, T q via an estimate of P p S (cid:16) 1 | x age q and, based on this, we approximate P p S (cid:16)
1 q by averaging over x age , i. e.,n
(cid:176) ni (cid:16) 1 P p S
i
(cid:16) 1 |
x age ,i q .
(2) For calibrating Γ , we take the maximum of the odds-ratio in (11) with the above estimates for P p S (cid:16) 1 q and P p S (cid:16) 1 | x age q plugged in, which yields a value for Γ
We use this data-driven calibration procedure in our experiments (Section 5).
In case the uncertainty regarding the calibration parameters remains high, large values for Γ can be chosen, yielding a wide uncertainty set and conservative policies.
OPTIMIZING GENERALIZABLE POLICIES
In this section, we derive an efficient algorithm for optimizing the minimax policy value in (16).
For this, we consider a parameterized policy class Π (cid:16) t π p(cid:4) , θ q : θ P Θ u and the minimax problem
[Equation]
where ψ i p θ q denotes ψ
i p π p(cid:4) , θ qq and corresponds to either (8), (9), or (10).
The above minimax problem is non-trivial, since it is in general non-convex in θ .
We first derive a closed-form solution of the worst-case policy value subproblem
(Section 4.1).
Then, based on this, we develop a tailored convex-concave procedure that solves (MMP)
(Section 4.2).
4.1 CLOSED-FORM SOLUTION OF WORST-CASE POLICY VALUE
The solution of (MMP) involves the worst-case policy value subproblem in (15).
We derive a closed-form solution of the subproblem and the corresponding Radon-Nikodým derivative at the optimal solution in Theorem 2.
Theorem 2.
(Closed-form solution of worst-case policy value.)
Let p i q denote the ordering such that ψ p 1 q p θ q ⁄ . . .
⁄ ψ p n q p θ q .
Then, an optimal solution of the worst-case policy value subproblem (15) is given by
[Equation]
[Equation]
with
[Equation]
The Radon-Nikodým derivative at optimal solution is given by R p i q
[Equation]
q [Equation] k (cid:6) u .
See Appendix A.3 for a proof.
Theorem 2 is appealing for two reasons: (i)
We prove that V Target p π ; R q is efficiently solved by a linear search over the sorted data.
(ii)
We prove that the worst-case policy value is given by a maximum over a finite set, which we use in the following section to show that the minimax problem can be written as a difference-ofconvex functions (DC) problem.
Based on this, we develop a convex-concave procedure to efficiently solve the minimax problem in (MMP).
4.2 MINIMAX CONVEX-CONCAVE PROCEDURE
In this section, we develop the m ini m ax c onvex- c oncave p rocedure (MMCCP) to efficiently solve the minimax problem (MMP).
For this, we derive a DC-representation of the worst-case policy value based on its closed-form solution in
Theorem 2.
For this, the following assumptions are made.
Assumption 1.
The set Θ is nonempty, compact, and convex.
Furthermore, π is a DC-function in θ , i. e., π p X, θ q (cid:16) ˜ g p X, θ q (cid:1) ˜
h p X, θ q , where ˜ g and ˜ h are convex in θ , and differentiable.
Note that Assumption 1 is very general as the class of DCfunctions is very rich.
For instance, it includes all twice continuously differentiable functions [Horst and Thoai, 1999].
We later show that Assumption 1 is fulfilled for the established policy class of logistic policies.
First, we show that ψ i p θ q can be written as a DC-function.
Lemma 1.
(DC-representation of ψ i p θ q .)
Under Assumption 1, ψ i p θ q is a DC-function in θ , i. e.,
[Equation]
where g i and h i are convex in θ .
See Appendix A.4 for a proof.
Now, using Lemma 1 and
Theorem 2, we prove that the worst-case policy value can be written as a DC-function.
Theorem 3.
(DC-representation of worst-case policy value.)
Under Assumption 1, the worst-case policy value
V Target p π ;
[Equation]
is [Equation] DC
[Equation]
function in
[Equation]
where g p θ q and h p θ q are convex and given by
[Equation]
with g i and h i from Lemma 1, and non-negative constants c i for all i .
See Appendix A.5 for a proof.
Finally, with the DCrepresentation of the worst-case policy value in Theorem 3, we can write the original minimax problem in (MMP) as a
DC-program, i. e.,
[Equation]
where g p θ q and h p θ q are convex and given in Theorem 3.
Hence, we can solve the minimax problem via a convexconcave procedure [Sriperumbudur and Lanckriet, 2009,
Yuille and Rangarajan, 2003].
This yields our tailored
MMCCP for solving (MMP) as outlined in Algorithm 1.
Next, we prove that the sequence p θ k q k P N generated by
MMCCP yields monotonically decreasing worst-case policy values and converges under mild assumptions.
Theorem 4.
(Theoretical Analysis of MMCCP.)
Suppose the outcomes are bounded, i. e., | Y | (cid:160) C , and Assumptionholds.
Then, the following holds true:
Input: Initial theta [Equation] , convergence tolerance
[Equation]
tol
Set [Equation]
—repeat
Solve the convex problem:
[Equation]
Set k [Equation] k [Equation]
until [Equation]
The sequence p θ k q k P N generated by MMCCP satisfies the monotonic descent property, i. e., for all k P N ,
[Equation]
If ˜ g and ˜ h from Assumption 1 are strongly convex, 5 then every limit point θ (cid:6) of p θ k q k P N is a stationary point 6 of ( MMP ).
Furthermore, it holds: lim k (cid:209)
[Equation] [Equation]
See Appendix A.6 for a proof.
To summarize, we develop a tailored convex-concave procedure that efficiently solves (MMP).
This is only possible since we proved that the worstcase policy value has a DC-representation (see Theorem 3).
In particular, our algorithm can be used on a rich class of policies and converges under mild assumptions.
We now demonstrate that Assumption 1 holds for an established parameterized policy class which we use in our experiments.
Logistic policies: Logistic policies are defined by π p X, θ q (cid:16) σ p θ ⊺ X q , where σ p z q (cid:16) 1 {p 1 (cid:0) e (cid:1) z q .
To find a DC-representation, it is sufficient to decompose σ p z q .
Hence, we set z
[Equation]
θ ⊺ X and write
[Equation]
It is straightforward to check that both functions are convex.
They can be made strongly convex by adding λ 2 z 2 to both functions.
Since ˜ g Log and ˜ h Log are differentiable, Assumption 1 is fulfilled and, hence, MMCCP converges for logistic policies.
In Appendix C, we show that Assumption 1 also holds for linear policies.
In addition, logistic policies also satisfy the generalization bound in Theorem 1, since they have ?
n -vanishing Rademacher complexity.
This can be seen by using that σ is Lipschitz together with the
Rademacher bound for linear classes [Maurer, 2006] and the scalar concentration inequality [Maurer, 2016].
EXPERIMENTS
In this section, we compare several policy learning methods to policies learned with our framework on the example of logistic policies.
We demonstrate that our framework generalizes substantially better to the target population.
5.1 SIMULATION STUDY
We first consider a simulation study to demonstrate the effect of unrepresentative training data.
For this, we consider the following data-generating process for the target population:
[Equation]
where m
[Equation]
Bern p 1 { 2 q , and ϵ (cid:18) N p 0 , 1 q .
The covariate mean is µ (cid:16) r - 1 , 1 { 2 , - 1 , 0 , - 1 s and the outcome means are β 0 (cid:16) r 0 , 3 { 4 , - 1 { 2 , 0 , - 1 s and β 1 (cid:16) r - 3 { 2 , 1 , - 3 { 2 , 1 , 1 { 2 s , respectively.
We sample from the target population using the following selection variable
[Equation]
which yields training data that is unrepresentative for the tail of the covariate distribution.
As baselines, we consider three established policy learning methods: the direct method ( DM ), the normalized inverse propensity weights method ( NIPW ), and the doubly robust method ( DR ).
We compare these established methods against our generalizable methods with each of the three ψ p θ q in (8), (9), and (10): the worst-case policy value obtained with the direct method ( GenDM ), obtained with the normalized IPW ( GenNIPW ), and with the doubly robust method ( GenDR ).
We use kernel and logistic regression for estimating µ t p x q and W IPW .
The parameter P p S (cid:16) 1 q is chosen by the data-driven calibration in Section 3.4 and Γ is varied across t 1 . 0 , 1 . 2 , 1 . 4 , 1 . 6 , 1 . 8 , 2 . 0 , 3 . 0 , . . .
, 10 . 0 u .
Details on implementation of MMCCP are in Appendix D.
We present the results for the different values of Γ in Figure 1.
Specifically, we show by how much our methods improve over the policy regret, V Target
[Equation]
q , of the corresponding baseline policy (i. e., DM, NIPW, and
DR) when tested on the target population.
Our methods achieve lower policy regrets on the target population across
[Figure]
    
[Figure]

[Figure]
HJ U H W,
[Figure]
U R Y
[Figure]
P
[Figure] W [Figure] all methods and across all values of Γ .
Specifically, relative to the policy regret of the baseline policy (green line), our methods (blue line) improve the policy regret on the target population by up to 40 %.
By construction, for Γ (cid:16) 1 (left end of plots), our methods resemble the baseline methods and yield the same policy regret.
When we increase Γ , our policies achieve substantial improvements of the policy regret on the target population over the baselines.
The best policy regret on the target population is achieved for Γ (cid:16) 8 , which is consistent with the simulation specifications, as the true Γ (cid:6) (cid:16) 8 .
For Γ (cid:16) 8 , relative to the baseline policies, our methods improve the policy regret by up to 40 %.
This demonstrates that policies learned with our framework generalize substantially better to the target population.
5.2 EXPERIMENTS ON CLINICAL TRIAL DATA
We evaluate our methods using the AIDS Clinical Trial
Group (ACTG) study
[Hammer et al., 1996], which is particularly suited for evaluating our framework.
This is because HIV-positive females tend to be underrepresented, which makes these studies not representative of the target population (i. e., the HIV-positive population in the USA)
[Gandhi et al., 2005, Greenblatt, 2011].
In fact, in the ACTG 175 study, only 5.8 % of the patients are female, whereas
HIV-positive females are more common in the USA population.
The outcome Y is the difference between the cluster of differentiation 4 (CD4) cell counts at the beginning of the study and the CD4 counts after 20 (cid:8) 5 weeks.
The average treatment effects on the male and female subgroups are -8.97 and -1.39, respectively, suggesting a large discrepancy in treatment effects between both subgroups.
We consider two treatment arms: one treatment arm for both zidovudine (ZDV) and zalcitabine (ZAL) ( T (cid:16) 1 ) vs. one treatment arm for ZDV only ( T (cid:16) 0 ), comprising 1 , 056 patients in total.
We consider 12 covariates (details on the covariates are in
Appendix B).
Again, we compare our methods against the established baseline methods.
This is a real-world clinical trial and, hence, we cannot access the true policy values on the target population.
However, we investigate the behavior of our policies by studying the percentage of patients that are treated (i. e., π p X q ¡ 0 . 5 ) for varying Γ .
For our GenDR, the result is presented in Figure 2.
The results for GenDM and GenNIPW are in
Appendix E.
We find that, compared to the baseline policy, our policy treats fewer patients for increasing Γ .
This seems reasonable, since females are underrepresented and have a lower average treatment effect.
Specifically, the baseline policy tends to treat more patients, since there are more patients in the study that benefit from the treatment.
However, in the target population (with a greater proportion of females), fewer patients are expected to benefit (due to the lower treatment effect in the female subgroup).
Our policy accounts for the underrepresentation of females and, as such, tends to treat fewer patients.
This result indicates the potential of our framework for learning policies that generalize to the target population.
CONCLUSION
We propose a novel framework for learning policies that generalize to the target population by optimizes the minimax policy value on the target population.
We prove that our framework yields policies that do not do worse on the target population than the worst-case policy value.
We solve the minimax problem via a tailored convex-concave procedure for which we prove convergence for parametrized spaces of policies.
Experiments demonstrate the benefit of learning generalizable policies using our framework.
[-q ` p 1 ´ π p X i qq µ 
-]
ACKNOWLEDGEMENTS
[-i q “ l 
-]
We thank the anonymous reviewers for valuable feedback.
This work was supported by the Swiss National Science
Foundation grants number 186932.