A DVERSARIAL
C OUNTERFACTUAL
E NVIRONMENT
M ODEL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
A good model for action-effect prediction, i.e., the environment model, is essential for sample-efficient policy learning, in which the agent can take numerous free trials to find good policies.
Currently, the model is commonly learned by fitting historical transition data through empirical risk minimization (ERM).
However, we discover that simple data fitting can lead to a model that will be totally wrong in guiding policy learning due to the selection bias in offline dataset collection.
In this work, we introduce weighted empirical risk minimization (WERM) to handle this problem in model learning.
A typical WERM method utilizes inverse propensity scores to re-weight the training data to approximate the target distribution.
However, during the policy training, the data distributions of the candidate policies can be various and unknown.
Thus, we propose an adversarial weighted empirical risk minimization (AWRM) objective that learns the model with respect to the worst case of the target distributions.
We implement AWRM in a sequential decision structure, resulting in the GALILEO model learning algorithm.
We also discover that GALILEO is closely related to adversarial model learning, explaining the empirical effectiveness of the latter.
We apply GALILEO in synthetic tasks and verify that GALILEO makes accurate predictions on counterfactual data.
We finally applied GALILEO in real-world offline policy learning tasks and found that
GALILEO significantly improves policy performance in real-world testing.
I NTRODUCTION
A good environment model is important for sample-efficient decision-making policy learning techniques like reinforcement learning (RL) (James & Johns, 2016).
The agent can take trials with this model to find better policies, then the costly real-world trial-and-errors can be saved (James & Johns, 2016; Yu et al., 2020) or completely waived (Shi et al., 2019).
In this process, the core of the models is to answer queries on counterfactual data unbiasedly, that is, given states, correctly answer what might happen if we were to carry out actions unseen in the training data (Levine et al., 2020).
Requiring counterfactual queries makes the environment model learning essentially different from standard supervised learning (SL) which directly fits the offline dataset.
In real-world applications, the offline data is often collected with selection bias, that is, for each state, each action might be chosen unfairly.
Seeing the example in Fig. 1(a), to keep the ball following a target line, a behavior policy will use a smaller force when the ball’s location is closer to the target line.
When a dataset is collected with selection bias, the association between the (location) states and (force) actions will make SL hard to identify the correct causal relationship of the states and actions to the next states respectively.
Then when we query the model with counterfactual data, the predictions might be catastrophic failures.
In Fig.
1(c), it mistakes that smaller forces will increase the ball’s next location.
Generally speaking, the problem corresponds to a challenge of training the model in one dataset but testing in another dataset with a shifted distribution (i.e., the dataset generated by counterfactual queries), which is beyond the SL’s capability as it violates the independent and identically distributed ( i.i.d. ) assumption.
The problem is widely discussed in causal inference for individual treatment effects (ITEs) estimation in many scenarios like patients’ treatment selection (Imbens, 1999; Alaa & van der Schaar, 2018).
ITEs are the effects of treatments on individuals, which are measured by treating each individual under a uniform policy and evaluate the effect differences.
Practical solutions use weighted empirical risk minimization (WERM) to handle this problem (Jung et al., 2020;
Shimodaira, 2000; Hassanpour & Greiner, 2019).
In particular, they estimate an inverse propensity score (IPS) to re-weight the training data to approximate the data distribution under a uniform policy.
Then a model is trained under the reweighted data distribution.
The distribution-shift problem is solved as ITEs estimation and model training are under the same distribution.
[Figure]
actions a t + ∆ a , where ∆ a ∈ [ − 1 , 1] is a variation of action value, the model of SL will exploit the association and give opposite responses, while in AWRM and its practical implementation GALILEO, the predictions are closer to the ground truths.
The result is in
Subfigure (c), where the darker a region is, the more samples are fallen in.
The selection bias can be regarded as an instance of the problem called “distributional shift” in offline model-based RL, which has also received great attention (Levine et al., 2020; Yu et al., 2020;
Kidambi et al., 2020; Chen et al., 2021).
However, previous methods, where naive supervised learning is used for environment model learning, ignore the problem in environment model learning to Reviewer rQ
[Figure]
The prediction risks is measured with mean square error (MSE).
The error of
SL is small only in training data ( ϕ = 62 . 5 ) but becomes much larger in the dataset “far away from” the training data.
AWRM-oracle selects the oracle worst counterfactual dataset for training for each iteration (pseudocode is in Alg. 1) which reaches small MSE in all datasets and gives correct response curves (Fig. 1(c)).
GALILEO approximates the optimal adversarial counterfactual data distribution based on the training data and model.
Although the
MSE of GALILEO is a bit larger than
SL in the training data, in the counterfactual datasets, the MSE is on the same scale as AWRM-oracle.
but handling the problem by suppressing the policy exploration and learning in risky regions.
Although these methods have made great progress in many tasks, so far, how to learn a better environment model that can alleviate the problem for faithful offline policy optimization has rarely been discussed.
In this work, for faithful offline policy optimization, we introduce WERM to environment model learning.
The extra challenge of model learning for policy optimization is that we have to query numerous different policies’ feedback besides the uniform policy for finding a good policy.
Thus the target data distribution to reweight can be various and unknown.
To solve the problem, we propose an objective called adversarial weighted empirical risk minimization (AWRM).
AWRM introduces adversarial policies, of which the corresponding counterfactual dataset has the maximal prediction error of the model.
For each iteration, the model is learned to be as small prediction risks as possible under the adversarial counterfactual dataset.
However, the adversarial counterfactual dataset cannot be obtained in the offline setting, thus we derive an approximation of the counterfactual data distribution queried by the optimal adversarial policy and use a variational representation to give a tractable solution to learn a model from the approximated data distribution.
As a result, we derive a practical approach named G enerative
A dversarial off LI ne counterfactua
L E nvironment m O del learning (GALILEO) for AWRM.
Fig.
2 shows the difference in the prediction errors learned by these algorithms.
We also discover that GALILEO is closely related to existing generative-adversarial model learning techniques, explaining the effectiveness of the latter.
Experiments are conducted in two synthetic and two realistic environments.
The results in the synthetic environments show that GALILEO can reconstruct correct responses for counterfactual queries.
The evaluation results in two realistic environments also demonstrate that GALILEO has better ability in counterfactual query compared with baselines.
We finally search for a policy based on the learned model in a real-world online platform.
The policy significantly improves performance in concerned business indicators.
R ELATED W ORK
We give related adversarial algorithms for model learning in the following and leave other related work in Appx.
F. GANTIE (Yoon et al., 2018) uses a generator to fill counterfactual outcomes for each data pair and a discriminator to judge the source (treatment group or control group) of the filled data pair.
The generator is trained to minimize the output of the discriminator.
GANITE is trained until the discriminator cannot determine which of the components is the factual outcome.
Bica et al.
(2020) propose SCIGAN to extend GANITE to continuous treatment effect estimation (a.k.a., dosage-response estimation) via a hierarchical discriminator architecture.
In real-world applications, environment model learning based on Generative Adversarial Imitation Learning (GAIL) has also been adopted for sequential decision-making problems (Ho & Ermon, 2016).
GAIL is first proposed for policy imitation (Ho & Ermon, 2016), which uses the imitated policy to generate trajectories by interacting with the environment.
The policy is learned with the trajectories through RL which maximizes the cumulative rewards given by the discriminator.
Shi et al.
; Chen et al.
Shang et al.
(2019) use GAIL for environment model learning by regarding the environment model as the generator and the behavior policy as the “environment” in standard GAIL.
These studies empirically demonstrate that adversarial model learning algorithms have better generalization ability for counterfactual queries, while our study reveals the connection between adversarial model learning and the WERM through IPS.
Our derived practical algorithm GALILEO is closely related to the existing adversarial model learning algorithms, explaining the effectiveness of the latter.
P RELIMINARIES 3. S INGLE - STEP I NDIVIDUALIZED T REATMENT E FFECTS E STIMATION AND W EIGHTED E MPIRICAL R ISKS M INIMIZATION
We first introduce individualized treatment effects (ITEs) estimation (Rosenbaum & Rubin, 1983), which can be regarded as the scenario in which the environment model has only a single step.
ITEs are typically defined as ITE ( x ) :
=
E [ M ∗ ( y | x, 1) | A = 1 , X = x ] − E
[ M ∗ ( y | x, 0)
|
A = 0 , X = x ] , where y is the feedback of the environment M ∗ ( y | x, a ) , X denotes the state vector containing pre-treatment covariates (such as age and weight), A denotes the treatment variable which is the action intervening to the state X , and A should be sampled from a uniform policy.
In the twotreatment scenario, A is in { 0 , 1 } where 1 is the action to intervene and 0 is the action to do nothing.
A correct ITEs estimation should be done in Randomized Controlled Trials (RCT) in which we have the same probability of samples of A = 1 and A = 0 for each X .
Here we use lowercase x, a and y to denote samples of random variables X, A and Y , and use X , A and
Y to denote space of the samples.
In practice, we prefer to estimate ITEs under observational studies.
In observational studies, datasets are pre-collected from the real world by a behavior policy such as a human-expert policy.
In this case, a common approach for estimating the ITEs can be ˆ ITE ( x i )
= a i ( y Fi − M ( x i , 1 − a i ))
+ (1 − a i )( M ( x i , 1 − a i ) − y Fi )
(Shalit et al., 2017) in deterministic prediction, where x i and y
Fi denote the covariate and factual feedback of the i -th sample, and M ∈ M denotes an approximated feedback model.
M is the space of the model.
In this formulation, the training set is an empirical factual data distribution P F = { ( x i , a i ) } ni and the testing set is an empirical counterfactual data distribution P CF =
{ ( x i , 1 − a i ) } ni .
If a does not sample from a discrete uniform policy, i.e., the policy has selection bias, P F and P CF will be two different distributions, which violate the i.i.d.
assumption of standard supervised learning.
In stochastic prediction, ˆ ITE ( x ) =
E [ M ( y | x, 1)] − E
[ M ( y | x, 0)] and the counterfactual distribution for testing is the dataset with action sampling from a uniform policy.
Generally speaking, in ITEs estimation, the risks of queries under counterfactual data are caused by the gap between the policy in training and testing data distributions.
Without further processing, minimizing the empirical risks cannot guarantee the counterfactual-query risks being minimized.
Assuming that the policy in training data µ satisfies µ ( a | x ) > 0 , ∀ a ∈ A , ∀ x ∈ X (often named overlap assumption), a classical solution to handle the above problem is weighted empirical risk minimization (WERM) through an inverse propensity scoring (IPS) term ω (Shimodaira, 2000;
Assaad et al., 2021; Hassanpour & Greiner, 2019; Jung et al., 2020):
Definition 3.1.
The learning objective of WERM through IPS is formulated as
[Equation]
where β and µ denote the policies in testing and training domains, and p µM ∗ is the joint probability p µM ∗ ( x, a, y ) :
= ρ 0 ( x ) µ ( a | x ) M ∗ ( y | x, a ) in which ρ 0 ( x ) is the distribution of state.
M is the model space.
ℓ is a loss function for model learning.
The ω is also known as importance sampling (IS) weight, which corrects the sampling bias.
In this objective, ω is to align the training data distribution to the testing data.
By selecting different ˆ ω to approximate ω to learn the model M , current environment model learning algorithms for ITEs estimation are fallen into the framework.
In standard supervised learning and some works for ITEs estimation (Wager & Athey, 2018; Weiss et al., 2015), ˆ ω ( x, a ) = 1 as the distribution-shift problem is ignored.
In Shimodaira (2000); Assaad et al.
; Hassanpour & Greiner (2019), ω = 1ˆ µ (i.e., β a uniform policy) for balancing treatment and control group, where ˆ µ is an approximation of behavior policy µ .
Note that it is a reasonable weight in ITEs estimation: ITEs are defined to evaluate the effect of each state between treatment and control behavior under a uniform policy.
3. S EQUENTIAL D ECISION - MAKING S ETTING
Decision-making processes in a sequential environment are often formulated into Markov Decision
Process (MDP) (Sutton & Barto, 1998).
MDP depicts an agent interacting with the environment through actions.
In the first step, states are sampled from an initial state distribution x 0 ∼ ρ 0 ( x ) .
Then at each time-step t ∈ { 0 , 1 , 2 , ... } , the agent takes an action a t ∈
A through a policy π ( a t | x t ) ∈
Π based on the state x t ∈ X , then the agent receives a reward r t from a reward function r ( x t , a t ) ∈ R and transits to the next state x t +1 given by a transition function M ∗ ( x t +1 | x t , a t ) built in the environment.
Π , X , and A denote the policy, state, and action spaces.
M ETHOD
In this section, we first propose a new offline model-learning objective based on Def. 3.1 for policy optimization tasks in Sec. 4.1;
In Sec. 4.2, we derive a tractable solution to the proposed objective;
Finally, we give a practical implementation in Sec. 4.3.
4. P ROBLEM F ORMULATION
For offline policy optimization, we require the environment model to have generalization ability in counterfactual queries since we need to query numerous different policies’ correct feedback from M .
Referring to the formulation of WERM through IPS in Def. 3.1, policy optimization requires M to minimize counterfactual-query risks under numerous unknown different policies rather than a specific target policy β .
More specifically, the question is: If β is unknown and can be varied, how should we reduce the risks in counterfactual queries?
In this article, we call the model learning problem in this setting “counterfactual environment model learning” and propose a new objective to handle the problem.
To be compatible with multi-step environment model learning, we first define a generalized
WERM through IPS based on Def. 3.1.
Definition 4.1.
Given the MDP transition function M ∗ that satisfies M ∗ ( x ′ | x, a ) > 0 , ∀ x ∈ X , ∀ a ∈
A , ∀ x ′ ∈ X and µ satisfies µ ( a | x ) > 0 , ∀ a ∈ A , ∀ x ∈ X , the learning objective of generalized
WERM through IPS is formulated as
[Equation]
s [Equation] where ρ µM ∗ is the training data distribution (collected by policy µ ), ρ βM ∗ is the testing data distribution (collected by policy β ).
We define ℓ M ( x, a, x ′ ) :
=
ℓ ( M ( x ′ | x, a ) , x ′ ) for brevity.
In an MDP, given any policy π , ρ πM ∗ ( x, a, x ′ )
=
ρ πM ∗ ( x ) π ( a | x ) M ∗ ( x ′ | x, a ) where ρ πM ∗ ( x ) denotes the occupancy measure of x for policy π , which can be defined as ρ πM ∗ ( x ) :=
(1 − γ )
E x 0 ∼ ρ
[ (cid:80) ∞ t =
γ t Pr( x t = x | x 0 , M ∗ )] (Sutton & Barto, 1998; Ho & Ermon, 2016) where
Pr π [ x t = x | x 0 , M ∗ ] is the state visitation probability that the policy π visits x at time-step t by executing in the environment M ∗ and starting at the state x 0 , and γ ∈ [0 , 1] is the discount factor.
Here we also define ρ πM ∗ ( x, a ) :
= ρ πM ∗ ( x ) π ( a | x ) for simplicity.
In this definition, ω can be rewritten as: ω ( x, a, x ′ )
=
ρ βM ∗ ( x ) β ( a | x ) M ∗ ( x ′ | x,a ) ρ µM ∗ ( x ) µ ( a | x ) M ∗ ( x ′ | x,a ) =
ρ βM ∗ ( x,a ) ρ µM ∗ ( x,a ) .
In single-step environments, for any policy π , ρ πM ∗ ( x ) = ρ 0 ( x ) .
Then we have ω ( x, a, x ′ )
=
ρ 0 ( x ) β ( a | x ) ρ 0 ( x ) µ ( a | x ) =
β ( a | x ) µ ( a | x ) , and the objective is degraded to
Eq. 1.
Therefore, Def. 3.1 is a special case of this generalized form.
Since β for counterfactual queries is unknown and can be varied in policy optimization, to reduce the risks in counterfactual queries in this scenario, we introduce adversarial policies which can induce the worst performance of the model predictions and propose to optimize WERM under the adversarial policies.
In particular, we propose A dversarial W eighted empirical R isk M inimization (AWRM) based on Def.
4.1 to handle this problem.
Definition 4.2.
Given the MDP transition function M ∗ , the learning objective of adversarial Weighted empirical risk minimization through IPS is formulated as
[Equation]
where the re-weighting term ω ( x, a | ρ βM ∗ ) is conditioned on the data distribution ρ βM ∗ of the adversarial policy β .
In the following, we will ignore ρ βM ∗ and use ω ( x, a ) for brevity.
According to the definition of MDP, ω ( x, a, x ′ )
=
ω ( x, a ) since the transition probability in the ratio can be canceled.
Eq.
3 minimizes the maximum model loss under all counterfactual data distributions ρ βM ∗ ( x, a, x ′ ) , β ∈ Π to guarantee the generalization ability for counterfactual data queried by policies in Π .
4. T RACTABLE AWRM S OLUTION
In this section, we propose a tractable solution to optimize Eq. 3.
The full derivations can be found in the Appx.
A.
We choose ℓ M to be the negative log-likelihood loss and derive the solution to Eq. 3:
[Equation]
where E M ∗ [ · ] denotes E x ′ ∼ M ∗ ( x ′ | x,a ) [ · ] .
The core problem is how to construct the data distribution ρ β ∗ M ∗ of the best-response policy β ∗ in M ∗ as it is costly to get extra data from M ∗ in real-world applications.
Instead of deriving the optimal β ∗ , our solution is to offline estimate the optimal adversarial distribution ρ β ∗ M ∗ with respect to M , then we can construct a surrogate objective to optimize M without directly querying the real environment M ∗ .
4.2. O PTIMAL A DVERSARIAL D ATA D ISTRIBUTION A PPROXIMATION
Ideally, given any M , it is obvious that the optimal β is the one that makes ρ βM ∗ ( x, a ) assign all densities to the point that has the largest negative log-likelihood.
However, searching for the maximum is impractical, especially in continuous space.
To give a relaxed but tractable solution, we add an Lregularizer to the original objective Eq. 3:
[Equation]
where α denotes the regularization coefficient of ρ βM ∗ and ∥ ρ βM ∗ ( · , · )
{+∥ 
+}
= (cid:82) X , A ( ρ βM ∗ ( x, a )) d a d x .
Then we can approximate the optimal distribution ρ ¯ β ∗ M ∗ via Lemma. 4.3.
Lemma 4.3.
Given any M in ¯ L ( ρ β M ∗ , M ) , the distribution of the ideal best-response policy ¯ β ∗ satisfies:
[Equation]
where D KL ( M ∗ ( ·| x, a ) , M ( ·| x, a )) is the Kullback-Leibler (KL) divergence between M ∗ ( ·| x, a ) and M ( ·| x, a ) , H M ∗ ( x, a ) denotes the entropy of M ∗ ( ·| x, a ) , and α M is the regularization coefficient α in Eq.
4 and also as a normalizer of Eq.
Intuitively, ρ ¯ β ∗ M ∗ has larger densities on the data where the divergence between the approximation model and the real model (i.e., D KL ( M ∗ ( ·| x, a ) , M ( ·| x, a )) ) is larger or the stochasticity of the real model (i.e., H M ∗ ) is larger.
However, the integral process of D KL in Eq.
5 is intractable in the offline setting as it explicitly requires the conditional probability function of M ∗ .
Our solution to solve the problem is utilizing the offline dataset D real as the empirical joint distribution ρ µM ∗ ( x, a, x ′ ) and adopting practical techniques for distance estimation on two joint distributions, like GAN (Goodfellow et al., 2014; Nowozin et al., 2016), to approximate Eq. 5.
To adopt that solution, we should first transform Eq.
5 into a form under joint distributions.
Without loss of generality, we introduce an intermediary policy κ , of which µ can be regarded as a specific instance.
Then we have M ( x ′ | x, a ) = to Reviewer bcJS ρ κM ( x, a, x ′ ) /ρ
κM ( x, a ) for any M if ρ κM ( x, a ) > 0 .
Assuming ∀ x ∈ X , ∀ a ∈ A , ρ κM ∗ ( x, a ) > 0 if ρ ¯ [Equation] M ∗ ( x, a ) > 0 , which will hold when κ overlaps with µ , then Eq.
5 can transform to:
[Equation]
where α 0 ( x, a )
=
α M ρ κM ∗ ( x, a ) .
We notice that the form ρ κM ∗ log ρ κM ∗ ρ κM is the integrated function in reverse KL divergence, which is an instance of f function in f -divergence (Ali & Silvey, 1966).
Replacing that form with f function, we obtain a generalized representation of ρ ¯ β ∗ M ∗ : ¯
[Equation]
where f : R + → R is a convex and lower semi-continuous (l.s.c.) function.
¯ ρ ¯ β ∗ M ∗ gives a generalized representation of the optimal adversarial data distribution to maximize the error of the model.
Based on Eq.
6, we have a surrogate objective of AWRM which can avoid querying M ∗ to construct ρ β ∗ M ∗ :
Theorem 4.4.
Let ¯ ρ ¯ β ∗ M ∗ as the data distribution of the best-response policy ¯ β ∗ in Eq.
4 under model
M θ parameterized by θ , then we can find the optimal θ ∗ of min θ max β ∈ Π ¯ L ( ρ βM ∗ , M θ ) (Eq. 4) via iteratively optimizing the objective θ t +1 =
min θ ¯ L (¯ ρ ¯ β ∗ M ∗ , M θ ) , where ¯ ρ ¯ β ∗ M ∗ is approximated via the last-iteration model M θ t .
Based on Corollary A.7, we derive an upper bound objective for min θ ¯ L (¯ ρ ¯ β ∗ M ∗ , M θ ) :
[Equation]
(cid:32) f (cid:32) ρ κM θt ( x, a, x ′ ) ρ κM ∗ ( x, a, x ′ ) (cid:33) − f (cid:32) ρ κM θt ( x, a ) ρ κM ∗ ( x, a ) (cid:33) + H M ∗ ( x, a ) (cid:33) (cid:124) (cid:123)(cid:122) (cid:125) W ( x,a,x ′ ) (cid:35) , where E ρ κM ∗ [ · ] denotes E x,a,x ′ ∼ ρ κM ∗ [ · ] , f is a l.s.c function satisfying f ′ ( x ) ≤ 0 , ∀ x ∈ X , and α 0 ( x, a ) =
α M θt ρ κM ∗ ( x, a ) .
Thm.
4.4 approximately achieve AWRM by using κ and a pseudo-reweighting module W .
W assigns learning propensities for data points with larger differences between distributions ρ κM θt and ρ κM ∗ .
By adjusting the weights, the learning process will exploit subtle errors in any data point, whatever how many proportions it contributes, to correct potential generalization errors on counterfactual data.
4.2. T RACTABLE S OLUTION TO T HM . 4.
In Thm. 4.4, the term f ( ρ κMθt ( x,a,x ′ ) ρ κM ∗ ( x,a,x ′ ) ) − f ( ρ κMθt ( x,a ) ρ κM ∗ ( x,a ) ) is still intractable.
To solve the problem, first, we resort to the first-order approximation of f .
Given some u ∈ (1 − ξ, 1 + ξ ) , ξ > 0 , we have
[Equation]
(7) where f ′ is the first-order derivative of f .
By Taylor’s formula and the fact that f ′ ( u ) of the generator function f is bounded in (1 − ξ, 1 + ξ ) , the approximation error is no more than O ( ξ 2 ) .
Let u = p ( x ) q ( x ) in Eq.
7, the pattern f
( p ( x ) q ( x ) ) in Thm.
4.4 can be converted to f ′ ( p ( x ) q ( x ) )
( p ( x ) q ( x ) − 1) + f (1) .
p ( x ) q ( x ) can be approximated by sampling from datasets and f ′ ( p ( x ) q ( x ) ) can be approximated by the corresponding variational representation T φ ∗ according to Lemma A.9 (Nowozin et al., 2016)
(see Appendix for details).
Based on any specific f -function, we can represent T φ with a discriminator
D φ .
However, Thm. 4.4 holds only when ∀ x ∈ X , f ′ ( x ) ≤ 0 .
It can be verified that the instance f ( u ) =
u log u − ( u + 1) log( u + 1) , T φ ( u ) =
log D φ ( u ) satisfy this condition (see Tab. 2 in the
Appendix).
We select that instance for the tractable solution.
In summary, by computing f ( · ) with first-order approximation (Eq. 7) and leveraging the variational representation T φ ∗ = log D φ ∗ to approximate f ′ ( ρ κMθt ρ κM ∗ ) , we can optimize the surrogate objective in Thm. 4.4 via:
[Equation]
where A φ ∗ 0 ,φ
[Equation]
1 ( x, a, x ′ ) =
log D φ [Equation] 0 ( x, a, x ′ )
− log D φ ∗ 1 ( x, a ) , E ρ κM [ · ] is a simplification of E x,a,x ′ ∼ ρ κM [ · ] and φ 0 and φ 1 are the parameters of T φ 0 and T φ 1 .
Based on Lemma A.9, we have f ′ ( ρ κMθt ( x,a,x ′ )
ρ κM ∗ ( x,a,x ′ ) )
≈ log D φ ∗ 0 ( x, a, x ′ ) and f ′ ( ρ κMθt ( x,a ) ρ κM ∗ ( x,a ) )
≈ log D φ ∗ 1 ( x, a ) .
Note that in the process, we ignore the term α 0 ( x, a ) for simplifying the objective.
The discussion on the impacts of removing α 0 ( x, a ) is left in App.
B.
In Eq.
8, the pseudo-reweighting module W in Thm.
4.4 is split into two terms in the RHS of the equation.
The first term is a generative adversarial training objective regarding M θ as the generator, while the second term is WERM through H M ∗ − A φ ∗ 0 ,φ ∗ 1 .
4. P RACTICAL I MPLEMENTATION
To give a practical implementation of the solution, extra two assumptions introduced in the process of modeling should be handled:
First , the approximation of Eq.
7 holds only when p ( x ) q ( x ) is close to 1 , which might not be satisfied.
To handle the problem, we inject a naive supervised learning loss and replace the second term of the objective Eq. 8 when the expectation of D φ 0 in ρ κM θt is far away from 0 .
5 ( f ′ (1) = log 0 . 5 );
Second , the overlap assumption of κ .
In practice, we need to use the real-world data to construct the distribution ρ κM ∗ and the generative data to construct ρ κM θt .
In the offline model-learning setting, we only have a real-world dataset D collected by the behavior policy µ .
We can learn a policy ˆ µ ≈ µ via behavior cloning based on D (Pomerleau, 1991; Ho & Ermon, 2016) and let ˆ µ be the policy κ .
Then we can regard D as the empirical data distribution of ρ κM ∗ and the trajectories to Reviewer bcJS collected by ˆ µ in the model M θ t as the empirical data distribution of ρ κM θt .
But the assumption ∀ x ∈ X , ∀ a ∈ A , µ ( a | x ) > 0 might not be satisfied.
To handle the problem, in behavior cloning, we model ˆ µ with a Gaussian distribution and constrain the lower bound of the variance with a small value ϵ µ > 0 to keep the assumption holding.
Besides, we add small Gaussian noises N (0 , ϵ D ) to the inputs of D φ to handle the mismatch between ρ µM ∗ and ρ ˆ µM ∗ due to ϵ µ . In Eq. 8, H M ∗ is unknown in advance.
In practice, we use H M θ to estimate it.
More specifically, the neural network of M θ can be modeled with a Gaussian distribution.
The variance of the Gaussian distribution is modeled with global variables Σ for each dimension of output.
We estimate H M ∗ with the closed-form solution of Gaussian entropy through Σ . to Reviewer bcJS Based on the above techniques, we propose G enerative A dversarial off LI
ne counterfactua
L
E nvironment m O del learning (GALILEO) for environment model learning.
GALILEO can be adopted in both single-step and sequential environment model learning.
The detailed implementation and comparison to previous adversarial methods are in Appx.
E.
E XPERIMENTS
In this section, we first conduct experiments in synthetic environments (Bica et al., 2020) to verify GALILEO on counterfactual queries 1 and the compatibility of GALILEO in sequential and single-step environments.
We select Mean Integrated Square Error MISE =
E (cid:104)(cid:82) A ( M ∗ ( y | x, a ) − M
( y | x, a ))
2 d a (cid:105) as the metric, which is a commonly used metric to measure the accuracy in counterfactual queries by considering the prediction errors in the whole action space.
Then we analyze the benefits of the implementation techniques described in Sec.
4.3 and [Figure] r oo t M I SE
GALILEO SL
IPW SCIGAN
[Figure] [Figure] the problems without them. Finally, we deploy GALILEO in two complex environments: MuJoCo in Gym (Todorov et al., 2012) and a real-world food-delivery platform to test the performance of
GALILEO in difficult tasks.
The algorithms compared are: (1) Supervised Learning (SL) : using standard empirical risk minimization for model learning; (2) Inverse Propensity Weighting (IPW)
(Spirtes, 2010): a standard implementation of WERM based IPS; (3) eStimating the effects of Continuous Interventions using GANs (SCIGAN)
(Bica et al., 2020): an adversarial algorithms for model learning used for causal effect estimation, which can be roughly regarded as a partial implementation of GALILEO (Refer to Appx. E.2).
We give a detailed description in Appx.
G.2.
to Reviewer rQ
5. T EST IN S YNTHETIC E NVIRONMENTS Test
in sequential environments Since there does not exist a task specifically designed for sequential environment model learning with selection bias, we construct a task, General Negative
Feedback Control (GNFC), which can represent a classic type of task with policies having selection bias.
Fig.
1(a) is also an example of GNFC.
We give detailed motivation, the effect of selection bias, and other details in Appx.
G.1.1.
We construct tasks on GNFC by adding behavior policies µ with different scales of uniform noise U ( − e, e ) with different probabilities p .
In particular, with e ∈ { 1 . 0 , 0 . 2 , 0 . 05 } and p ∈ { 1 . 0 , 0 . 2 , 0 . 05 } , we construct 9 tasks and name them with the format of “e*_p*”.
For example, e1_p0.2 is the task with behavior policy injecting with U ( − 1 , 1) with 0.
probability.
The results of GNFC tasks are summarized in Fig. 3(a) and the detailed results can be found in Tab. 8.
The results show that the property of the behavior policy (i.e., e and p ) dominates the generalization ability of the baseline algorithms.
When e = 0 .
05 , almost all of the baselines fail and give a completely opposite response curve (see Fig. 4(a) and Appx.
H.2).
IPW still perform well when 0 .
2 ≤ e ≤ 1 .
but fails when e = 0 .
05 , p < = 0 .
We also found that SCIGAN can reach a better performance than other baselines
when e = 0 .
05 , p < = 0 .
2 , but the results in other tasks are unstable.
GALILEO is the only algorithm that is robust to the selection bias and outputs correct response curves in all of the tasks.
Based on the experiment, we also indicate that the commonly used overlap assumption is unreasonable to a certain extent especially in real-world applications since it is impractical to inject noises into the whole action space.
The problem of overlap assumption being violated should be taken into consideration otherwise the algorithm will be hard to use in practice if it is sensitive to the noise range.
[Figure]
Test in single-step environments Previous experiments on counterfactual environment model learning are based on single-step semi-synthetic data simulation (Bica et al., 2020).
Since GALILEO is compatible with single-step environment model learning, we select the same task named
TCGA in Bica et al.
(2020) to test GALILEO.
Based on three synthetic response functions in TCGA, we construct 9 tasks by choosing different parameters of selection bias on µ which is constructed with beta distribution, and design a coefficient c to control the selection bias of the beta distribution.
We name the tasks with the format of “t?_bias?”.
For example, t1_bias2 is the task with the first response functions and c = 2 .
The detail of TCGA is in Appx.
G.1.2.
The results of TCGA tasks are summarized in Fig. 3(b) and the detailed results can be found in Tab. 9 in Appendix.
We found the phenomenon in this experiment is similar to the one in GNFC, which demonstrates the compatibility of GALILEO to single-step environments.
We also found that the results of IPW are unstable in this experiment.
It might be because the behavior policy is modeled with beta distribution while the propensity score ˆ µ is modeled with Gaussian distribution.
Since IPW directly reweight loss function with 1 ˆ µ , the results are sensitive [Table] to the error on ˆ µ .
GALILEO also models ˆ µ with Gaussian distribution but the results are more stable since GALILEO does not re-weight through ˆ µ explicitly.
Response curve visualization We plot the averaged response curves which are constructed by equidistantly sampling action from the action space and averaging the feedback of the states in the dataset as the averaged response.
Parts of the results in Fig.
4 (all curves can be seen in Appx.
H.2).
For those tasks where baselines fail in reconstructing response curves, GALILEO not only reaches a better MISE score but reconstructs almost exact responses.
Ablation studies In Sec.
4.3, we introduce several techniques to develop a practical GALILEO algorithm.
Based on task e0.2_p0.05 of GNFC, we give the ablation studies to investigate the effects of these techniques.
As the main-body space is limited, we leave the results in Appx.
H.3.
T EST IN C OMPLEX E NVIRONMENTS
In MuJoCo tasks MuJoCo is a benchmark task in Gym (Todorov et al., 2012; Brockman et al., 2016) where we need to control a robot with specific dynamics to complete some tasks (e.g., standing or running).
We select 3 environment from D4RL (Fu et al., 2020) to construct our model learning tasks.
We compare it with a standard transition model learning algorithm used in the previous offline model-based RL algorithms (Yu et al., 2020; Kidambi et al., 2020), which is a variant of supervised learning.
We name the method OFF-SL.
Besides, we also implement IPW and SCIGAN as the baselines.
In D4RL benchmark, only the “medium” tasks is collected with a fixed policy, i.e., the behavior policy is with 1/3 performance to the expert policy), which is most matching to our proposed problem.
So we train models in datasets HalfCheetah-medium, Walker2d-medium, and Hopper-medium. We trained the models with the same gradient steps and saved the models. to Reviewer bcJS, j5p5, and rQ79 We first verify the generalization ability of the models by adopting them into offline model-based
RL. Instead of designing sophisticated tricks to suppress policy exploration and learning in risky regions as current offline model-based RL algorithms (Yu et al., 2020; Kidambi et al., 2020) do, we just use the standard SAC algorithm Haarnoja et al. (2018) to exploit the models for policy learning to strictly verify the ability of the models. Unfortunately, we found that the compounding error will still be inevitably large in the 1,000-step rollout, which is the standard horizon in MuJoCo tasks, leading all models to fail to derive a reasonable policy.
To better verify the effects of models on policy optimization, we learn and evaluate the policies with three smaller horizons: H ∈ { 10 , 20 , 40 } .
The results are listed in Tab. 1.
We first averaged the normalized return (refer to “avg. norm.”) under each task, and we can see that the policy obtained by GALILEO is significantly higher than other models (the improvements are 24% to 161%). At the same time, we found that SCIGAN performed better in policy learning, while IPW performed similarly to SL. This is in line with our expectations, since IPW only considers the uniform policy as the target policy for debiasing, while policy optimization requires querying a wide variety of policies. Minimizing the prediction risks only under a uniform policy cannot yield a good environment model for policy optimization.
Besides, in IPW, the cumulative effects of policy on the state distribution are ignored.
On the other hand, SCIGAN, as a partial implementation of GALILEO (refer to Appx. E.2), also roughly achieves AWRM and considers the cumulative effects of policy on the state distribution, so its overall performance is better; In addition, we find that GALILEO achieves significant improvement in 6 of the 9 tasks. But in HalfCheetah, IPW works slightly better. However, compared with MAX-RETURN, it can be found that all methods fail to derive reasonable policies because their policies’ performances are far away from the optimal policy.
By further visualizing the trajectories, we found that all the learned policies just keep the cheetah standing in the same place or even going backward.
This phenomenon is also similar to the results in MOPO (Yu et al., 2020).
In MOPO’s experiment in the medium datasets, the truncated-rollout horizon used in Walker and Hopper for policy training is set to 5, while HalfCheetah has to be set to the minimal value: 1 .
These phenomena indicate that HalfCheetah may still have unknown problems, resulting in the generalization bottleneck of [Figure] [Figure] r e sc a l ed a v e r aged r e s pon s e
GALILEOSL (a) response curves in City A 0.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
to 1)
[Figure]
[Figure]
[Figure] r [Figure] sc a l ed c u m u l a t
i v e t r ea t m en t e ff e c t s randomGALILEOSL
[Figure]
[Figure]
number
[Figure]
[Figure] [Figure] r [Figure]
sc a l ed f i v e - m i nu t e o r de r- t a k en r a t e
[Figure] models. In a real-world platform We finally deploy GALILEO in a real-world large-scale food-delivery platform.
The goal of the platform is to balance the demand from take-out food orders and the supply of delivery clerks, i.e., helping delivery clerks fulfill more orders by giving reasonable strategies.
We focus on a Budget Allocation task to the Time period (BAT) in the platform (see Appx.
G.1.3 for details).
The goal of the BAT task is to handle the imbalance problem between the demanded orders from customers and the supply of delivery clerks in different time periods by allocating reasonable allowances to those time periods. The core challenge of the environment model learning in BAT tasks is similar to the challenge in Fig. Specifically, the behavior policy in BAT tasks is a human-expert policy, which tends to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise tends to decrease the budget (Fig. 12 gives a real-data instance of this phenomenon).
We first learn a model to predict the supply of delivery clerks (measured by fulfilled order amount) on given allowances. Although the SL model can efficiently fit the offline data, the tendency of the response curve is easily to be incorrect. As can be seen in Fig. 5(a), with a larger budget of allowance, the prediction of the supply is decreased in SL, which obviously goes against our prior knowledge.
This is because, in the offline dataset, the corresponding supply will be smaller when the allowance is larger. It is conceivable that if we learn a policy through the model of SL, the optimal solution is canceling all of the allowances, which is obviously incorrect in practice. On the other hand, the tendency of GALILEO’s response is correct. Fig.
13 plots all the results in 6 cities. to Reviewer rQ and j5p5 Second, we conduct randomized controlled trials (RCT) in one of the testing cities.
Using the RCT samples, we can evaluate the generalization ability of the model predictions via Area Under the Uplift Curve (AUUC) (Betlei et al., 2020), which measure the correctness of the sort order of the model prediction in RCT samples.
The AUUC further show that GALILEO gives a reasonable sort order on the supply prediction (see Fig. 5(b)) while the standard SL technique fails to complete this task.
Finally, we search for the optimal policy via the cross-entropy method planner (Hafner et al., 2019) based on the learned model and deploy the policy in a real-world platform. The results of A/B test in
City A is shown in Fig. 5(c).
It can be seen that after the day of the A/B test, the treatment group (deploying our policy) significant improve the five-minute order-taken rate than the baseline policy (the same as the behavior policy).
In summary, the policy improves the supply from 0.14 to 1.
percentage points to the behavior policies in the 6 cities .
The details of these results are in Appx.
H.6.
D ISCUSSION AND F UTURE W ORK
In this work, we propose AWRM which handles the generalization challenges of the counterfactual environment model learning.
By theoretical modeling, we give a tractable solution to handle AWRM and propose GALILEO.
GALILEO is verified in synthetic environments, complex robot control tasks, and a real-world platform, and shows great generalization ability on counterfactual queries.
Giving correct answers to counterfactual queries is important for policy learning.
We hope the work can inspire researchers to develop more powerful tools for counterfactual environment model learning.
The current limitation lies in: There are several simplifications in the theoretical modeling process (further discussion is in Appx.
B), which can be modeled more elaborately .
Besides, experiments on
MuJoCo indicate that these tasks are still challenging to give correct predictions on counterfactual data.
These should also be further investigated in future work.
Appendix Table of Contents A Proof of Theoretical Results
A.1 Proof of Lemma 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Proof of Eq.
A.3 Proof of Thm. 4.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Proof of the Tractable Solution . . . . . . . . . . . . . . . . . . . . . . . . . .
B Discussion of the Theoretical Results
C Societal Impact
D AWRM-oracle Pseudocode
E Implementation
E.1 Details of the GALILEO Implementation . . . . . . . . . . . . . . . . . . . . .
E.2 Connection with Previous Adversarial Algorithms . . . . . . . . . . . . . . . .
F Additional Related Work
G Experiment Details
G.1 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.2 Baseline Algorithms . .
G.3 Hyper-parameters . .
G.4 Computation Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H Additional Results
H.1 All of the Result Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
A P ROOF OF T HEORETICAL R ESULTS
The overall pipeline to model the tractable solution to AWRM is given in Fig.
In the proof section, we replace the notation of E with an integral for brevity.
Now we rewrite the original objective ¯ L ( ρ βM ∗ , M ) as:
[Equation]
where ω ( x, a ) = ρ β M ∗ ( x,a ) ρ µM ∗ ( x,a ) and ∥ ρ βM ∗ ( · , · ) ∥
=
(cid:82) X , A ρ βM ∗ ( x, a ) 2 d a d x , which is the squared l 2 -norm.
In an MDP, given any policy π , ρ πM ∗ ( x, a, x ′ )
=
ρ πM ∗ ( x ) π ( a | x ) M ∗ ( x ′ | x, a ) where ρ πM ∗ ( x ) denotes the occupancy measure of x for policy π , which can be defined (Sutton & Barto, 1998; Ho & Ermon, 2016) as ρ πM ∗ ( x ) :=
(1 − γ )
E x 0 ∼ ρ
[ (cid:80) ∞ t =
γ t Pr( x t = x | x 0 , M ∗ )] where Pr π [ x t = x | x 0 , M ∗ ] is the state visitation probability that π starts at state x 0 in model M ∗ and receive x at timestep t and γ ∈ [0 , 1] is the discount factor.
[Figure]
A. P ROOF OF L EMMA 4.
For better readability, we first rewrite Lemma 4.3 as follows:
Lemma A.1.
Given any M in ¯ L ( ρ βM ∗ , M ) , the distribution of the ideal best-response policy ¯ β ∗ satisfies:
[Equation]
[Equation]
where D KL ( M ∗ ( ·| x, a ) , M ( ·| x, a )) is the Kullback-Leibler (KL) divergence between M ∗ ( ·| x, a ) and M ( ·| x, a ) , H M ∗ ( x, a ) denotes the entropy of M ∗ ( ·| x, a ) , where D KL ( M ∗ ( ·| x, a ) , M ( ·| x, a )) is the Kullback-Leibler (KL) divergence between M ∗ ( ·| x, a ) and M ( ·| x, a ) , H M ∗ ( x, a ) denotes the entropy of M ∗ ( ·| x, a ) , and α M is the regularization coefficient α in Eq.
9 and also as a normalizer.
Proof.
Given a transition function M of an MDP, the distribution of the best-response policy β ∗ satisfies:
[Equation]
− g ( · , · ) α ∥ 22 .
We know that the occupancy measure ρ βM ∗ is a density function with a constraint (cid:82)
X (cid:82) A ρ βM ∗ ( x, a )d a d x = 1 .
Assuming the occupancy measure ρ βM ∗ has an upper bound c , that is 0 ≤ ρ βM ∗ ( x, a ) ≤ c, ∀ a ∈ A , ∀ x ∈ X , constructing a regularization coefficient α M = (cid:82)
X (cid:82) A ( D KL ( M ∗ ( ·| x, a ) , M ( ·| x, a ))
+ H M ∗ ( x, a ))
d x d a as a constant value given any M , then we have
[Equation]
which is the optimal density function of Eq.
9 with α = α M .
Note that in some particular M ∗ , we still cannot construct a β that can generate an occupancy specified by g ( x, a ) /α
M for any M .
We can only claim the distribution of the ideal best-response policy ¯ β ∗ satisfies:
[Equation]
where α M is a normalizer that α M = (cid:82) X (cid:82) A ( D KL ( M ∗ ( ·| x, a ) , M ( ·| x, a ))
+ H M ∗ ( x, a ))
d x d a .
We give a discussion of the rationality of the ideal best-response policy ¯ β ∗ as a replacement of the real best-response policy β ∗ in Remark A.2.
Remark A.2.
The optimal solution Eq.
11 relies on g ( x, a ) .
In some particular M ∗ , it is intractable to derive a β that can generate an occupancy specified by g ( x, a ) /α
M .
Consider the following case: a state x 1 in M ∗ might be harder to reach than another state x 2 , e.g., M ∗ ( x 1 | x, a ) <
M ∗ ( x 2 | x, a ) , ∀ x ∈ X , ∀ a ∈ A , then it is impossible to find a β that the occupancy satisfies ρ βM ∗ ( x 1 , a ) > ρ βM ∗ ( x 2 , a ) .
In this case, Eq.
11 can be a sub-optimal solution.
Since this work focuses on task-agnostic solution derivation while the solution to the above problem should rely on the specific description of M ∗ , we leave it as future work.
However, we point out that Eq.
11 is a reasonable re-weighting term even as a sub-optimum: ρ ¯ β ∗ M ∗ gives larger densities on the data where the distribution distance between the approximation model and the real model (i.e., D KL ( M ∗ , M ) ) is larger or the stochasticity of the real model (i.e., H M ∗ ) is larger.
A. P ROOF OF E Q .
The integral process of D KL in Eq.
5 is intractable in the offline setting as it explicitly requires the conditional probability function of M ∗ .
Our motivation for the tractable solution is utilizing the offline dataset D real as the empirical joint distribution ρ µ M ∗ ( x, a, x ′ ) and adopting practical techniques for distance estimation on two joint distributions, like GAN (Goodfellow et al., 2014; Nowozin et al., 2016), to approximate Eq. 5.
To adopt that solution, we should first transform Eq.
5 into a form under joint distributions.
Without loss of generality, we introduce an intermediary policy κ , of which µ can be regarded as a specific instance.
Then we have M ( x ′ | x, a )
= ρ κM ( x, a, x ′ ) /ρ
κM ( x, a ) for any M if ρ κM ( x, a ) > 0 .
Assuming ∀ x ∈ X , ∀ a ∈ A , ρ κM ∗ ( x, a ) > 0 if ρ ¯ β ∗ M ∗ ( x, a ) > 0 , which will hold when κ overlaps with µ , then Eq.
5 can transform to:
[Equation]
where α 0 ( x, a )
=
α M ρ κM ∗ ( x, a ) .
Definition A.3 ( f -divergence) .
Given two distributions P and Q , two absolutely continuous density functions p and q with respect to a base measure d x defined on the domain X , we define the f -divergence (Nowozin et al., 2016),
[Equation]
where the generator function f : R + → R is a convex, lower-semicontinuous function.
We notice that the terms ρ κM ∗ ( x, a, x ′ ) log ρ κM ∗ ( x,a,x ′ )
ρ κM ( x,a,x ′ ) and ρ κM ∗ ( x, a ) log ρ κM ∗ ( x,a ) ρ κM ( x,a ) are the integrated functions in reverse KL divergence, which is an instance of f function in f -divergence (See Reverse- KL divergence of Tab.1 in (Nowozin et al., 2016) for more details).
Replacing that form q log qp with qf ( pq ) , we obtain a generalized representation of ρ ¯ β ∗ M ∗ : ¯ ρ ¯ β ∗ M ∗ := [Equation]
[Equation]
κ
M
[Equation]
(cid:33)
[Equation]
A. P ROOF OF T HM . 4.
We first introduce several useful lemmas for the proof.
Lemma A.4.
Rearrangement inequality
The rearrangement inequality states that, for two sequences a 1 ≥ a 2 ≥ . . .
≥ a n and b 1 ≥ b 2 ≥ . .
≥ b n , the inequalities a 1 b 1 + a 2 b 2 + · · · + a n b n ≥ a 1 b π (1) + a 2 b π (2) + · · · + a n b π ( n ) ≥ a 1 b n + a 2 b n − 1 + · · · + a n bhold, where π (1) , π (2) , . .
π ( n ) is any permutation of 1 , 2 , . . . , n .
Lemma A.5.
For two sequences a 1 ≥ a 2 ≥ . . . ≥ a n and b 1 ≥ b 2 ≥ . . .
≥ b n , the inequalities n (cid:88) i =
[Equation]
(cid:88)
[Equation] [Equation]
hold.
Proof.
By rearrangement inequality, we have n (cid:88)
[Equation]
=
a
[Equation]
n b n n (cid:88) i
= a i b i ≥ a 1 b 2 + a 2 b 3 + · · ·
[Equation]
a n bn (cid:88) i = a i b i ≥ a 1 b 3 + a 2 b 4 + · · · + a n b... n (cid:88) i
=
a i b i ≥ a 1 b n + a 2 b 1 + · · · + a n b n −
Then we have n n (cid:88) i =
a i b i ≥ n (cid:88) i =
a i n (cid:88) i =
b i n (cid:88) i =  na i b i ≥ n (cid:88) i =  na i (cid:88)
1 nb
i
Now we extend Lemma A.5 into the continuous integral scenario:
Lemma A.6.
Given X ⊂ R , for two functions f : X → R and g : X → R that f ( x ) ≥ f ( y ) if and only if g ( x ) ≥ g ( y ) , ∀ x, y ∈ X , the inequality
[Equation]
x holds, where p : X → R and p ( x ) > 0 , ∀ x ∈ X and (cid:82) X p ( x )d x = 1 .
Proof.
Since ( f ( x ) − f ( y ))
( g ( x ) − g
y )) ≥ 0 , ∀ x,
y ∈ X , we have
[Equation]
[Table]
Corollary A.7.
Let g ( p ( x ) q ( x ) )
= − log p ( x ) q ( x ) where p ( x ) > 0 , ∀ x ∈ X and q ( x ) > 0 , ∀ x ∈ X , for υ > 0 , the inequality
[Equation]
holds if f ′ ( x ) ≤ 0 , ∀ x ∈ X .
It is not always satisfied for f functions of f -divergence.
We list a comparison of f on that condition in Tab. 2.
Proof.
g ′ ( x ) = − log x
= − 1 x < 0 , ∀ x ∈ X .
Suppose f ′ ( x ) ≤ 0 , ∀ x ∈ X , we have f ( x ) ≥ f ( y ) if and only if g ( x ) ≥ g ( y ) , ∀ x, y ∈ X holds.
Thus f ( υ p ( x ) q ( x ) ) ≥ f ( υ p ( y ) q ( y ) )
if and only if g ( p ( x ) q ( x ) ) ≥ g ( p ( y ) q ( y ) ) , ∀ x, y ∈ X holds for all υ > 0 .
By defining F ( x ) =
f ( υ p ( x ) q ( x ) )) and G ( x ) =
g ( p ( x ) q ( x ) ) and using Lemma A.6, we have:
[Equation]
Then we know
[Equation]
[Table]
holds.
[Table]
Now, we prove Thm. 4.4.
For better readability, we first rewrite Thm. 4.4 as follows:
Theorem A.8.
Let ¯ ρ ¯ β ∗ M ∗ as the data distribution of the best-response policy ¯ β ∗ in Eq.
4 under model M θ parameterized by θ , then we can find the optimal θ ∗ of min θ max β ∈ Π ¯ L ( ρ βM ∗ , M θ ) (Eq. 4) via iteratively optimizing the objective θ t +1 =
min θ ¯ L (¯ ρ ¯ β ∗ M ∗ , M θ ) , where ¯ ρ ¯ β ∗ M ∗ is approximated via the last-iteration model M θ t .
Based on Corollary A.7, we have an upper bound objective for min θ ¯ L (¯ ρ ¯ β ∗ M ∗ , M θ ) and derive the following objective
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) W ( x,a,x ′ ) (cid:35) , where α 0 ( x, a ) =
α M θt ρ κM ∗ ( x, a ) , E ρ κM ∗ [ · ] denotes E x,a,x ′ ∼ ρ κM ∗ [ · ] , f is the generator function in f -divergence which satisfies f ′ ( x ) ≤ 0 , ∀ x ∈ X , and θ is the parameters of M .
M θ t denotes a probability function with the same parameters as the learned model (i.e., ¯ θ = θ ) but the parameter is fixed and only used for sampling.
Proof.
Let ¯ ρ ¯ β ∗ M ∗ as the data distribution of the best-response policy ¯ β ∗ in Eq.
4 under model M θ parameterized by θ , then we can find the optimal θ t +1 of min θ max β ∈ Π ¯ L ( ρ βM ∗ , M θ )
(Eq. 4) via iteratively optimizing the objective θ t +1 =
min θ ¯ L (¯ ρ ¯ β ∗ M ∗ , M θ ) , where ¯ ρ ¯ β ∗ M ∗ is approximated via the last-iteration model M θ t :
[Equation]
[Table]
( − log M θ ( x ′ | x, a )) (cid:33) d x ′ − ρ κM ∗ ( x, a ) (cid:32) f
(cid:32) ρ κM θt ( x, a ) ρ κM ∗ ( x, a ) (cid:33) − H M ∗ ( x, a ) (cid:33) (cid:90)
X M ∗ ( x ′ | x, a )( − log M θ ( x ′ | x, a ))
d x ′ (cid:33) d a d x = max θ (cid:90)
X , A , X α 0 ( x, a ) ρ κM ∗ ( x, a, x ′ ) log M θ ( x ′ | x, a ) (cid:32) f
(cid:32) ρ κM θt ( x, a, x ′ )
ρ κM ∗ ( x, a, x ′ ) (cid:33) − f (cid:32) ρ κM θt ( x, a ) ρ κM ∗ ( x, a ) (cid:33) + H M ∗ ( x, a ) (cid:33)
d
x ′ d a d x, (17) where M θ t is introduced to approximate the term ¯ ρ ¯ β ∗ M ∗ and fixed when optimizing θ .
In Eq.
16, ∥ ρ βM ∗ ( · , · ) ∥ 22 for Eq.
9 is eliminated as it does not contribute to the gradient of θ .
Assume f ′ ( x ) ≤ 0 , ∀ x ∈ X , let υ ( x, a ) :
= ρ κMθt ( x,a ) ρ κM ∗ ( x,a ) > 0 , p ( x ′ | x, a ) =
M θ ( x ′ | x, a ) , and q ( x ′ | x, a ) =
M ∗ ( x ′ | x, a ) , the first inequality can be derived by adopting Corollary A.7 and eliminating the first H M ∗ since it does not contribute to the gradient of θ .
A. P ROOF OF THE T RACTABLE S OLUTION
Now we are ready to prove the tractable solution:
Proof.
The core challenge is that the term f ( ρ κMθt ( x,a,x ′ ) ρ κM ∗ ( x,a,x ′ ) )
− f ( ρ κMθt ( x,a ) ρ κM ∗ ( x,a ) ) is still intractable.
In the following, we give a tractable solution to Thm. 4.4.
First, we resort to the first-order approximation.
Given some u ∈ (1 − ξ, 1 + ξ ) , ξ > 0 , we have
[Equation]
(18) where f ′ is the first-order derivative of f .
By Taylor’s formula and the fact that f ′ ( u ) of the generator function f is bounded in (1 − ξ, 1+ ξ ) , the approximation error is no more than O ( ξ 2 ) .
Substituting u with p ( x ) q ( x ) in Eq.
18, the pattern f
( p ( x ) q ( x ) ) in Eq.
17 can be converted to p ( x ) q ( x ) f ′ ( p ( x ) q ( x ) )
− f ′ ( p ( x ) q ( x ) )
+ f (1) , then we have:
[Equation]
[Table]
Note that the part ρ κM ∗ ( x, a ) in ρ κM ∗ ( x, a, x ′ ) can be canceled because of α 0 ( x, a )
=
α M θt ρ κM ∗ ( x, a ) , but we choose to keep it and ignore α 0 ( x, a ) .
The benefit is that we can estimate ρ κM ∗ ( x, a, x ′ ) from an empirical data distribution through data collected by κ in M ∗ directly, rather than from a uniform distribution which is harder to be generated.
Although keeping ρ κM ∗ ( x, a ) incurs extra bias in theory, the results in our experiments show that it has not made significant negative effects in practice.
We leave this part of modeling in future work.
In particular, by ignoring α 0 ( x, a ) , we have:
[Equation]
We can estimate f ′ (cid:18)
ρ κMθt ( x,a ) ρ κM ∗ ( x,a ) (cid:19) and f ′ (cid:18) ρ κMθt ( x,a,x
[Equation]
κM
[Equation]
( x
[Equation]
a
[Equation]
x
[Equation]
through Lemma A.9.
Lemma A.9 ( f ′ ( pq ) estimation (Nguyen et al., 2010)) .
Given a function T φ : X → R parameterized by φ ∈ Φ , if f is convex and lower semi-continuous, by finding the maximum point of φ in the following objective:
[Equation]
we have f ′ ( p ( x ) q ( x ) )
=
T φ ∗ ( x ) .
f ∗ is Fenchel conjugate of f (Hiriart-Urruty & Lemaréchal, 2001).
In particular,
[Equation]
then we have f ′ (cid:18)
[Equation]
κMθt ( x,a,x
[Equation]
x,a,x
[Equation]
T
[Equation]
( x, a, x ′ ) and f ′ (cid:18)
[Equation]
( x,
[Equation]
ρ κM ∗ ( x,
[Equation]
(cid:19)
≈ T
φ ∗ 1 ( x, a ) .
Given φ ∗ 0 and φ ∗ 1 , let A φ ∗ 0 ,φ ∗ 1 ( x, a, x ′ )
=
T φ ∗ 0 ( x, a, x ′ ) − T
φ
∗ 1 ( x, a ) , then we can optimize θ via:
[Equation]
Based on the specific f -divergence, we can represent T and f ∗ ( T ) with a discriminator D φ .
It can be verified that f ( u ) =
u log u − ( u + 1) log( u + 1) , T φ ( u ) =
log D φ ( u ) , and f ∗ ( T φ ( u ))
=
− log(1 − D φ ( u )) proposed in Nowozin et al.
(2016) satisfies the condition f ′ ( x ) ≤ 0 , ∀ x ∈ X
(see
Tab. 2).
We select the former in the implementation and convert the tractable solution to: θ t +1 = arg max θ
[Equation] [Equation]
[Equation]
21) where A φ ∗ 0 ,φ ∗ 1 ( x, a, x ′ ) =
log D φ ∗ 0 ( x, a, x ′ )
− log D φ ∗ 1 ( x, a ) , E ρ κMθt [ · ] is a simplification of
E x,a,x ′ ∼ ρ κMθt [ · ] .
B D ISCUSSION OF THE T HEORETICAL R ESULTS
We summarize the limitations of current theoretical results and future work as follows:
As discussed in Remark A.2, the solution Eq.
11 relies on ρ βM ∗ ( x, a ) ∈ [0 , c ] , ∀ a ∈ A , ∀ x ∈ X .
In some particular M ∗ , it is intractable to derive a β that can generate an occupancy specified by g ( x, a ) /α
M .
If more knowledge of M ∗ or β ∗ is provided or some mild assumptions can be made on the properties of M ∗ or β ∗ , we may model ρ in a more sophisticated approach to alleviating the above problem.
In the tractable solution derivation, we ignore the term α 0 ( x, a ) =
α M θt ρ κ M ∗ ( x, a ) (See Eq. 20).
The benefit is that ρ κM ∗ ( x, a, x ′ ) in the tractable solution can be estimated through offline datasets directly.
Although the results in our experiments show that it does not produce significant negative effects in these tasks, ignoring ρ κM ∗ ( x, a ) indeed incurs extra bias in theory.
In future work, techniques for estimating ρ κM ∗ ( x, a ) (Liu et al., 2020) can be incorporated to correct the bias.
On the other hand, α M θt is also ignored in the process.
α M θt can be regarded as a global rescaling term of the final objective Eq. 20.
Intuitively, it constructs an adaptive learning rate for Eq. 20, which increases the step size when the model is better fitted and decreases the step size otherwise.
It can be considered to further improve the learning process in future work, e.g., cooperating with empirical risk minimization by balancing the weights of the two objectives through α M θt .
C S OCIETAL I MPACT
This work studies a method toward counterfactual environment model learning.
Reconstructing an accurate environment of the real world will promote the wide adoption of decision-making policy optimization methods in real life, enhancing our daily experience.
We are aware that decision-making policy in some domains like recommendation systems that interact with customers may have risks of causing price discrimination and misleading customers if inappropriately used.
A promising way to reduce the risk is to introduce fairness into policy optimization and rules to constrain the actions (Also see our policy design in Sec.
G.1.3).
We are involved in and advocating research in such directions.
We believe that business organizations would like to embrace fair systems that can ultimately bring long-term financial benefits by providing a better user experience.
D AWRM- ORACLE P SEUDOCODE
We list the pseudocode of AWRM-oracle in Alg. 1.
Process :
E I MPLEMENTATION E. D ETAILS OF THE GALILEO I MPLEMENTATION
The approximation of Eq.
18 holds only when p ( x ) q ( x ) is close to 1 , which might not be satisfied.
To handle the problem, we inject a standard supervised learning loss
[Equation]
(22) to replace the second term of the above objective when the output probability of D is far away from 0 .
5 ( f ′ (1) = log 0 . 5 ).
In the offline model-learning setting, we only have a real-world dataset D collected by the behavior policy µ .
We learn a policy ˆ µ ≈ µ via behavior cloning with D (Pomerleau, 1991; Ho & Ermon, 2016) and let ˆ µ be the policy κ .
We regard D as the empirical data distribution of ρ κM ∗ and the trajectories collected by ˆ µ in the model M θ t as the empirical data distribution of ρ κM θt .
But the assumption ∀ x ∈ X , ∀ a ∈ A , µ ( a | x ) > 0 might not be satisfied.
In behavior cloning, we model ˆ µ with a Gaussian distribution and constrain the lower bound of the variance with a small value ϵ µ >to keep the assumption holding.
Besides, we add small Gaussian noises u ∼ N (0 , ϵ D ) to the inputs of D φ to handle the mismatch between ρ µM ∗ and ρ ˆ µM ∗ due to ϵ µ .
In particular, for φ 0 and φ 1 learning, we have:
[Equation]
where E ρ κMθt , u [ · ] is a simplification of E x,a,x ′ ∼ ρ κMθt , u ∼N (0 ,ϵ D ) [ · ] and u = [ u x , u a , u x ′ ] .
On the other hand, we notice that the first term in Eq.
21 is similar to the objective of
GAIL (Ho & Ermon, 2016) by regarding M θ as the policy to learn and κ as the environment to generate data.
For better capability in sequential environment model learning, here we introduce some practical tricks inspired by GAIL for model learning (Shi et al., 2019;
Shang et al., 2019): we introduce an MDP for κ and M θ , where the reward is defined by the discriminator D , i.e., r ( x, a, x ′ ) =
log D ( x, a, x ′ ) .
M θ is learned to maximize the cumulative rewards.
With advanced policy gradient methods (Schulman et al., 2015; 2017), the objective is converted to max θ (cid:2)
A φ ∗ 0 ,φ ∗ 1 ( x, a, x ′ ) log M θ ( x, a, x ′ ) (cid:3) , where A = Q κ M θt − V κ M θt , dataset
D via Eq
26 and Eq.
27 respectively ρ ˆ µM θt is estimated D gen ρ µM ∗ is estimated
Update Q and V via Eq.
24 and Eq.
through D gen , D φ 0 ,
D
Update the model M θ t via the first term of Eq. 23, which is implemented with a standard policy gradient method like TRPO ( et al
2015) or PPO (Schulman et al., 2017).
8: if p D gen [ D φ 0 ( t a t t
)] < p then 9:
Compute the gradient of M θ t via the second term of Eq. 23 and record it as g sl 10: else 11:
Compute the gradient of M via Eq.
22 and record it as 12: end
Rescale g sl via Eq
Update
15: end
Q κM ¯ θ ( x, a, x ′ )
=
E [ (cid:80) ∞ t =0 γ t r ( x t , a t , x t +1 ) | ( x t , a t , x t +1 ) =
( x, a, x ′ ) , κ, M θ t ] , and V κM ¯ θ ( x, a ) =
E M ¯ θ (cid:104) Q κM ¯ θ ( x, a, x ′ ) (cid:105) .
A in Eq.
21 can also be constructed similarly.
Although it looks unnecessary in theory since the one-step optimal model M θ is the global optimal model in this setting, the technique is helpful in practice as it makes A more sensitive to the compounding effect of one-step prediction errors: we would consider the cumulative effects of prediction errors induced by multi-step transitions in environments.
In particular, to consider the cumulative effects of prediction errors induced by multi-step of transitions in environments, we overwrite function A φ ∗ 0 ,φ ∗ 1 as A φ ∗ 0 ,φ ∗
= Q κM θt − V κM θt , where Q κM θt ( x, a, x ′ ) =
E (cid:2)(cid:80) ∞
t γ t log D φ ∗ 0 ( x t , a t , x t +1 ) | ( x t , a t , x t +1 ) =
( x, a, x ′ ) , κ, M θ t (cid:3) and
V κM θt ( x, a ) =
E (cid:2)(cid:80) ∞
t γ t log D φ ∗ 1 ( x t , a t ) | ( x t , a t ) =
( x, a ) , κ, M θ t (cid:3) .
To give an algorithm for singlestep environment model learning, we can just set γ in Q and V to 0 . [Figure]
By adopting the above implementation techniques [Equation] we convert the objective into the following formulation
[Equation]
A
[Equation]
In practice, GALILEO optimizes the first term of Eq.
23 with conservative policy gradient algorithms (e.g., PPO (Schulman et al., 2017) or TRPO (Schulman et al., 2015)) to avoid unreliable gradients for model improvements.
Eq.and
Eq. 27 are optimized with supervised learning.
The second term of Eq.
23 is optimized with supervised learning with a re-weighting term − A φ ∗ 0 ,φ ∗ 1 + H M ∗ .
Since H M ∗ is unknown, we use H M θ to estimate it.
When the mean output probability of a batch of data is larger than 0 .
6 or small than 0 .
4 , we replace the second term of Eq.
23 with a standard supervised learning in Eq.
Besides, unreliable gradients also exist in the process of optimizing the second term of Eq. 23.
In our implementation, we use the scale of policy gradients to constrain the gradients of the second term of Eq. 23.
In particular, we first compute the l 2 -norm of the gradient of the first term of Eq.via conservative policy gradient algorithms, named
|| g pg
|| 2 .
Then we compute the l 2 -norm of the gradient of the second term of Eq.
23, name ||
g sl || 2 .
Finally, we rescale the gradients of the second term g sl by
[Equation]
For each iteration, Eq. 23, Eq. 26, and Eq.
27 are trained with certain steps (See Tab. 5) following the same framework as GAIL.
Based on the above techniques, we summarize the pseudocode of
GALILEO in Alg. 2, where p 0 and p 1 are set to 0 .
4 and 0 .
in all of our experiments.
The overall architecture is shown in Fig.
to Reviewer bcJS
E.
C ONNECTION WITH P REVIOUS A DVERSARIAL A LGORITHMS
Standard GAN (Goodfellow et al., 2014) can be regarded as a partial implementation including the first term of Eq.
23 and Eq.
by degrading them into the single-step scenario.
In the context of
GALILEO, the objective of GAN is
[Equation]
where A φ ∗ ( x, a, x ′ )
=
log D φ ∗ ( x, a, x ′ ) .
In the single-step scenario, ρ κM θt ( x, a, x ′ ) =
ρ 0 ( x ) κ ( a | x ) M θ t ( x ′ | a, x ) .
The term E ρ κ Mθt [ A φ ∗ ( x, a, x ′ ) log M θ ( x ′ | x, a )] can convert to
E ρ κMθ [log D φ ∗ ( x, a, x ′ )] by replacing the gradient of M θ t ( x ′ | x, a ) ∇ θ log M θ ( x ′ | x, a ) with ∇ θ M θ ( x ′ | x, a ) (Sutton & Barto, 1998).
Previous algorithms like GANITE (Yoon et al., 2018) and SCIGAN (Bica et al., 2020) can be regarded as variants of the above training framework.
The first term of Eq. 23 and Eq.
26 are similar to the objective of GAIL by regarding M θ as the “policy” to imitate and ˆ µ as the “environment” to collect data.
In the context of GALILEO, the objective of GAIL is:
[Equation]
where A φ ∗ ( x, a, x ′ )
=
Q κM θ ( x, a, x ′ ) − V κM θ ( x, a ) and V κM θt ( x, a ) =
E M θt ( x,a )
[ Q κ ( x, a, x ′ )] .
F A DDITIONAL R ELATED W ORK
to Reviewer j5p and rQ79 Our primitive objective is inspired by weighted empirical risk minimization (WERM) based on inverse propensity score (IPS). WERM is originally proposed to solve the generalization problem of domain adaptation in machine learning literature. For instance, we would like to train a predictor
M ( y | x ) in a domain with distribution P train ( x ) to minimize the prediction risks in the domain with distribution P test ( x ) , where P test ̸ = P test . To solve the problem, we can train a weighted objective with max M E x ∼ P train [ P test ( x ) P train ( x ) log M ( y | x )] , which is called weighted empirical risk minimization methods (Ben-David et al., 2006; 2010; Cortes et al., 2010; Byrd & Lipton, 2019; Quinonero-Candela et al., 2008). These results have been extended and applied to causal inference, where the predictor is required to be generalized from the data distribution in observational studies (source domain) to the data distribution in randomized controlled trials (target domain)
(Shimodaira, 2000; Assaad et al., 2021; Hassanpour & Greiner, 2019; Jung et al., 2020; Johansson et al., 2018). In this case, the input features include a state x (a.k.a. covariates) and an action a (a.k.a. treatment variable) which is sampled from a policy. We often assume the distribution of x , P ( x ) is consistent between the source domain and the test domain, then we have P test ( x ) P train ( x ) =
P ( x ) β ( a | x ) P ( x ) µ ( a | x ) = β ( a | x ) µ ( a | x ) , where µ and β are the policies in source and target domains respectively. In Shimodaira (2000); Assaad et al. (2021); Hassanpour & Greiner (2019), the policy in randomized controlled trials is modeled as a uniform policy, then P test ( x ) P train ( x ) =
P ( x ) β ( a | x ) P ( x ) µ ( a | x ) = β ( a | x ) µ ( a | x ) ∝ 1 µ ( a | x ) . 1 µ ( a | x ) is also known as inverse propensity score (IPS). In Johansson et al. , it assumes that the policy in the target domain is predefined as β ( a | x ) before environment model learning, then it uses βµ as the IPS.
The differences between AWRM and previous works are fallen in two aspects: (1) We consider the distribution-shift problem in the sequential decision-making scenario. In this scenario, we not only consider the action distribution mismatching between the behavior policy µ and the policy to evaluation β , but also the follow-up effects of policies to the state distribution; (2) For faithful offline policy optimization, we require the environment model to have generalization ability in numerous different policies. The objective of AWRM is proposed to guarantee the generalization ability of M in numerous different policies instead of a specific policy.
On a different thread, there are also studies that bring counterfactual inference techniques of causal inference into model-based RL (Buesing et al., 2019; Pitis et al., 2020; Sontakke et al., 2021).
These works consider that the transition function is relevant to some hidden noise variables and use Pearl-style structural causal models (SCMs), which is a directed acyclic graphs to define the causality of nodes in an environment, to handle the problem. SCMs can help RL in different ways:
Buesing et al.
(2019) approximate the posterior of the noise variables based on the observation of data, and environment models are learned based on the inferred noises.
The generalization ability is improved if we can infer the correct value of the noise variables.
Pitis et al.
(2020) discover several local causal structural models of a global environment model, then data augmentation strategies by leveraging these local structures to generate counterfactual experiences.
Sontakke et al. (2021) proposes a representation learning technique for causal factors, which is an instance of the hidden noise variables, in partially observable Markov decision processes (POMDPs). With the learned representation of causal factors, the performance of policy learning and transfer in downstream tasks will be improved. Instead of considering the hidden noise variables in the environments, our to Reviewer j5p study considers the environment model learning problem in the fully observed setting and focuses on unbiased causal effect estimation in the offline dataset under behavior policies collected with selection bias.
In offline model-based RL, the problem is called distribution shift (Yu et al., 2020; Levine et al., 2020; Chen et al., 2021) which has received great attentions.
However, previous algorithms do not handle the model learning challenge directly but propose techniques to suppress policy sampling and learning in risky regions (Yu et al., 2020; Kidambi et al., 2020).
Although these algorithms have made great progress in offline policy optimization in many tasks, so far, how to learn a better environment model in this scenario has rarely been discussed.
G E XPERIMENT D ETAILS G. S ETTINGS G.1. G ENERAL N EGATIVE F EEDBACK C ONTROL (GNFC)
The design of GNFC is inspired by a classic type of scenario that behavior policies µ have selection bias and easily lead to counterfactual risks: For some internet platforms, we would like to allocate budgets to a set of targets (e.g., customers or cities) to increase the engagement of the targets in the platforms.
Our task is to train a model to predict targets’ feedback on engagement given targets’ features and allocated budgets.
In these tasks, for better benefits, the online working policy (i.e., the behavior policy) will tend to cut down the budgets if targets have better engagement, otherwise, the budgets might be increased.
The risk of counterfactual environment model learning in the task is that: the object with better historical engagement will be sent to smaller budgets because of the selection bias of the behavior policies, then the model might exploit this correlation for learning and get a conclusion that: increasing budgets will reduce the targets’ engagement, which violates the real causality.
We construct an environment and a behavior policy to mimic the above process.
In particular, the behavior policy µ GNFC is
[Equation]
where ϵ is a sample noise, which will be discussed later.
The environment includes two parts:
(1) response function M
[Equation]
y |
[Equation]
) mapping function M 2 ( x ′ | x, y ) :
[Equation]
The transition function M ∗ is a composite of M ∗ ( x ′ | x, a ) =
M 2 ( x ′ | x, a, M 1 ( y | x, a )) .
The behavior policies have selection bias: the actions taken are negatively correlated with the states, as illustrated in Fig. 8(a) and Fig. 8(b).
We control the difficulty of distinguishing the correct causality of x , a , and y by designing different strategies of noise sampling on ϵ .
In principle, with a larger number or more pronounced disturbances, there are more samples violating the correlation between x and a , then more samples can be used to find the correct causality.
Therefore, we can control the difficulty of counterfactual environment model learning by controlling the strength of disturbance.
In particular, we sample ϵ from a uniform distribution U ( − e, e ) with probability p .
That is, ϵ = 0 with probability 1 − p and ϵ ∼ U ( − e, e ) with probability p .
Then with larger p , there are more samples in the dataset violating the negative correlation (i.e., µ GNFC ), and with larger e , the difference of the feedback will be more obvious.
By selecting different e and p , we can construct different tasks to verify the effectiveness and ability of the counterfactual environment model learning algorithm.
G.1. T HE C ANCER G ENOMIC A TLAS (TCGA)
The Cancer Genomic
Atlas (TCGA) is a project that has profiled and analyzed large numbers of human tumors to discover molecular aberrations at the DNA, RNA, protein, and epigenetic levels.
The resulting rich data provide a significant opportunity to accelerate our understanding of the molecular basis of cancer.
We obtain features, x , from the TCGA dataset and consider three continuous treatments as done in SCIGAN (Bica et al., 2020).
Each treatment, a , is associated with a set of parameters, v 1 , v 2 , v 3 , that are sampled randomly by sampling a vector from a standard normal distribution and scaling it with its norm.
We assign interventions by sampling a treatment, a , from [Figure] a beta distribution, a | x ∼ Beta ( α, β ) .
α ≥ 1 controls the sampling bias and β =
α − 1 a ∗ + 2 − α , where a ∗ is the optimal treatment.
This setting of β ensures that the mode of Beta ( α, β ) is a ∗ .
The calculation of treatment response and optimal treatment are shown in Table [Table]
[Table]
We conduct experiments on three different treatments separately and change the value of bias α to assess the robustness of different methods to treatment bias.
When the bias of treatment is large, which means α is large, the training set contains data with a strong bias on treatment so it would be difficult for models to appropriately predict the treatment responses out of the distribution of training data.
G.1. B UDGET A LLOCATION TASK TO THE T IME PERIOD (BAT)
We deploy GALILEO in a real-world large-scale food-delivery platform.
The platform contains various food stores, and food delivery clerks.
The overall workflow is as follows: the platform presents the nearby food stores to the customers and the customers make orders, i.e., purchase take-out foods from some stores on the platform.
The food delivery clerks can select orders from the platform to fulfill.
After an order is selected to fulfill, the delivery clerks will take the ordered take-out foods from the stores and then send the food to the customers.
The platform will pay the delivery clerks (mainly in proportion to the distance between the store and the customers’ location) once the orders are fulfilled.
An illustration of the workflow can be found in Fig.
However, there is an imbalance problem between the demanded orders from customers and the supply of delivery clerks to fulfill these orders.
For example, at peak times like lunchtime, there will be many more demanded orders than at other times, and the existed delivery clerks might not be able to fulfill all of these orders timely.
The goal of the Budget Allocation task to the Time period (BAT) is to handle the imbalance problem in time periods by sending reasonable allowances to different time periods.
More precisely, the goal of BAT is to make all orders (i.e., the demand) sent in different time periods can be fulfilled (i.e., the supply) timely.
[Figure] To handle the imbalance problem in different time periods, in the platform, the orders in different time periods t ∈ [0 , 1 , 2 ..., 23] will be allocated with different allowances c ∈ N + .
For example, at 10 A.M. (i.e., t = 10 ), we add 0 .
5$ (i.e., c = 0 . 5 ) allowances to all of the demanded orders.
From 10 A.M. to 11 A.M., the delivery clerks who take orders and send food to customers will receive extra allowances.
Specifically, if the platform pays the delivery clerks 2$ for fulfilling the order, now he/she will receive 2 . 5$ .
For each day, the budget of allowance C is fixed.
We should find the best budget allocation policy π ∗ ( c | t ) of the limited budget C to make as many orders as possible can be taken timely.
To find the policy, we first learn a model to reconstruct the response of allowance for each delivery clerk ˆ M ( y t +1 | s t , p t , c t ) , where y t +1 is the taken orders of the delivery clerks in state s t , c t is the allowances, p t denotes static features of the time period t .
In particular, the state s t includes historical order-taken information of the delivery clerks, current orders information, the feature of weather, city information, and so on.
Then we use a rule-based mapping function f to fill the complete next time-period states, i.e., s t +
= f
( s t , p t , c t , y t +1 ) .
Here we define the composition of the above functions ˆ M and f as ˆ M f .
Finally, we learn a budget allocation policy based on the learned model.
For each day, the policy we would like to find is:
[Equation]
In our experiment, we evaluate the degree of balancing between demand and supply by computing the averaged five-minute order-taken rate, that is the percentage of orders picked up within five minutes.
Note that the behavior policy is fixed for the long term in this application.
So we directly use the data replay with a small scale of noise (See Tab. 5) to reconstruct the behavior policy for model learning in GALILEO.
Also note that although we model the response for each delivery clerk, for fairness, the budget allocation policy is just determining the allowance of each time period t and keeps the allowance to each delivery clerk s the same.
G. B ASELINE A LGORITHMS
The algorithm we compared are: (1) Supervised Learning (SL): training a environment model to minimize the expectation of prediction error, without considering the counterfactual risks; (2) inverse propensity weighting (IPW)
(Spirtes, 2010): a practical way to balance the selection bias by re-weighting.
It can be regarded as ω = 1 ˆ µ , where ˆ µ is another model learned to approximate the behavior policy; (3) SCIGAN:
a recent proposed adversarial algorithm for model learning for continuous-valued interventions (Bica et al., 2020).
All of the baselines algorithms are implemented with the same capacity of neural networks (See Tab. 5).
G.2. S UPERVISED L EARNING (SL)
As a baseline, we train a multilayer perceptron model to directly predict the response of different treatments, without considering the counterfactual risks.
We use mean square error to estimate the performance of our model so that the loss function can be expressed as MSE = 1 n (cid:80) ni =
( y i − ˆ y i ) 2 , where n is the number of samples, y is the true value of response and ˆ y is the predicted response.
In practice, we train our SL models using Adam optimizer and the initial learning rate 3 e − 4 on both datasets TCGA and GNFC.
The architecture of the neural networks is listed in Tab. 5.
G.2. I NVERSE P ROPENSITY W EIGHTING (IPW)
Inverse propensity weighting (Spirtes, 2010) is an approach where the treatment outcome model uses sample weights to balance the selection bias by re-weighting.
The weights are defined as the inverse propensity of actually getting the treatment, which can be expressed as 1 ˆ µ ( a | x ) , where x stands for the feature vectors in a dataset, a is the corresponding action and ˆ µ ( a | x ) indicates the action taken probability of a given the features x within the dataset.
ˆ µ is learned with standard supervised learning.
Standard IPW leads to large weights for the points with small sampling probabilities and finally makes the learning process unstable.
We solve the problem by clipping the propensity score: ˆ µ ← min(ˆ µ, 0 . 05) , which is common used in existing studies (Ionides, 2008).
The loss function can thus be expressed as 1 n (cid:80) ni
= 1 ˆ µ ( a i | x i ) ( y i − ˆ y i ) 2 .
The architecture of the neural networks is listed in
Tab. 5.
G.2. SCIGAN
SCIGAN (Bica et al., 2020) is a model that uses generative adversarial networks to learn the data distribution of the counterfactual outcomes and thus generate individualized response curves.
SCIGAN does not place any restrictions on the form of the treatment-does response functions and is capable of estimating patient outcomes for multiple treatments, each with an associated parameter.
SCIGAN first trains a generator to generate response curves for each sample within the training dataset.
The learned generator can then be used to train an inference network using standard supervised methods.
For fair comparison, we increase the number of parameters for the the open-source version of SCIGAN so that the SCIGAN model can have same order of magnitude of network parameters as GALILEO.
In addition, we also finetune the hyperparameters (Tab. 4) of the enlarged SCIGAN to realize its full strength.
We set num_dosage_samples 9 and λ = 10 .
[Table]
. H YPER - PARAMETERS
We list the hyper-parameter of GALILEO in Tab. 5.
G. C OMPUTATION R ESOURCES
We use one Tesla V100 PCIe 32GB GPU and a 32-core Intel(R)
Xeon(R)
Gold 5118 CPU @ 2.30GHz to train all of our model. [Table]
H A DDITIONAL R ESULTS H. A LL OF THE R ESULT T ABLE
We give the result of CNFC in Tab. 8, TCGA in Tab. 9, BAT in Tab. [Table] and MuJoCo in Tab. 6.
H. A VERAGED R ESPONSE C URVES
We give the averaged responses for all of the tasks and the algorithms in Fig.
16 to Fig.
We randomly select 20% of the states in the dataset and equidistantly sample actions from the action space for each sampled state [Table] and plot the averaged predicted feedback of each action.
The real response is slightly different among different figure as the randomly-selected states for testing is different.
We sample 9 points in GNFC tasks and 33 points in TAGC tasks for plotting.
H. A BLATION S TUDIES
[Table] [Figure] [Table] [Figure]
[Table]
O N E _ S T E
P S
I N G L E _ S L P
U R E _
G A I L
N O _
I N J E C T _ N O
I
S
E      s q r t M
I SE
In Sec.
4.3 and Appx.
E.1, we introduce several techniques to develop a practical GALILEO algorithm.
Based on task e0.2_p0.05 in GNFC, we give the ablation studies to investigate the effects of these techniques.
We first compare two variants that do not handle the assumptions violation problems: (1) NO_INJECT_NOISE : set ϵ µ and ϵ D to zero, which makes the overlap assumption not satisfied;; (2) SINGLE_SL : without replacing the second term in
Eq. 8 with standard supervised learning even when the output probability of D is far away from 0 .
Besides, we introduced several tricks inspired by GAIL and give a comparison of these tricks and GAIL: (3) ONE_STEP : use one-step reward instead of cumulative rewards (i.e., Q and V; see Eq.
24 and Eq. 25) for re-weighting, which is implemented by set γ to 0 ; (4) SINGE_DIS : remove T φ ∗ 1 ( x, a ) and replace it with E M θ (cid:2) T φ ∗ 0 ( x, a, x ′ ) (cid:3) , which is inspired by GAIL that uses a value function as a baseline instead of using another discriminator; (5) PURE_GAIL : remove the second term in Eq. 8.
It can be regarded as a naive adoption of GAIL and a partial implementation of GALILEO.
We summarize the results in Fig. 10.
Based on the results of NO_INJECT_NOISE and SINGLE_SL , we can see that handling the assumption violation problems is important and will increase the ability [Table] on counterfactual queries.
The results of PURE_GAIL tell us that the partial implementation of
GALILEO is not enough to give stable predictions on counterfactual data; On the other hand, the result of ONE_STEP also demonstrates that embedding the cumulative error of one-step prediction is helpful for GALILEO training; Finally, we also found that SINGLE_DIS nearly has almost no effect on the results.
It suggests that, empirically, we can use E M θ (cid:2) T φ ∗ 0 ( x, a, x ′ ) (cid:3) as a replacement for T φ ∗ 1 ( x, a ) , which can reduce the computation costs of the extra discriminator training.
H. W ORST -C ASE P REDICTION E
RROR
In theory, GALILEO increases the generalization ability by focusing on the worst-case samples’ training to achieve AWRM.
To demonstrate the property, we propose a new metric named Mean-Max
Square Error (MMSE): E (cid:104) max a ∈A ( M ∗ ( x ′ | x, a ) − M ( x ′ | x, a )) 2 (cid:105) and give the results of MMSE for GNFC in Tab. 10 and for TCGA in Tab. 11.
H. D ETAILED R ESULTS IN THE M U J O C O T ASKS
We select 3 environments from D4RL (Fu et al., 2020) to construct our model learning tasks.
We compare it with a typical transition model learning algorithm used in the previous offline model-based RL algorithms (Yu et al., 2020; Kidambi et al., 2020), which is a variant of standard supervised learning.
We name the method OFF-SL.
We train models in datasets HalfCheetah-medium, Walker2dmedium, and Hopper-medium, which are collected by a behavior policy with 1/3 performance to the expert policy, then we test them in the corresponding expert dataset.
We plot the converged results and learning curves of GALILEO and OFF-SL in three MuJoCo tasks in Tab. 6 and Fig.respectively.
In Fig.
11, we can see that both OFF-SL and GALILEO perform well in the training datasets.
OFF-SL can even reach a bit lower error in halfcheetah and walker2d.
However, when we verify the models through “expert” and “medium-replay” datasets, which are collected by other policies, the performance of GALILEO is significantly more stable and better than OFF-SL.
As the training continues, OFF-SL even gets worse and worse.
In summary, GALILEO reaches significantly better performances in the expert dataset: the averaged declines of root MSE in three environments are 56.5%, 49.2%, and 34.8%.
However, whether in GALILEO or OFF-SL, the performance for testing is at least 2x worse than in the training dataset.
The phenomenon indicates that although GALILEO can make better performances for counterfactual queries, the risks of using the models are still large and still challenging to be further solved.
[Figure]
[Figure]
[Figure]
n o r m a li z e d f u l f ill e d o r d e r a m o un t
H. D ETAILED R ESULTS IN THE BAT T ASK
The core challenge of the environment model learning in BAT tasks is similar to the challenge in
Fig.
Specifically, the behavior policy in BAT tasks is a human-expert policy, which will tend to increase the budget of allowance in the time periods with a lower supply of delivery clerks, otherwise will decrease the budget (Fig.
12 gives an instance of this phenomenon in the real data).
Since there is no oracle environment model for querying
[Figure]
we have to describe the results with other metrics
[Figure]
First [Figure] we review whether the tendency of the response curve is consistent.
In this application, with a larger budget of allowance, the supply will not be decreased.
As can be seen in Fig.
[Figure] the tendency of GALILEO’s response is valid in 6 cities but almost all of the models of SL give opposite directions to the response.
If we learn a policy through the model of SL, the optimal solution is canceling all of the allowances, which is obviously incorrect in practice.
Second [Figure] we conduct randomized controlled trials (RCT) in one of the testing cities.
Using the RCT samples, we can evaluate the correctness of the sort order of the model predictions via Area Under [Figure] [Figure] [Figure]
[Figure]
(rescaled to 1) 1. r e sc a l ed c u m u l a t i v e t r ea t m en t e ff e c t s randomGALILEOSL the Uplift Curve (AUUC)
(Betlei et al., 2020).
To plot AUUC, we first sort the RCT samples based on the predicted treatment effects.
Then the cumulative treatment effects are computed by scanning the sorted sample list.
If the sort order of the model predictions is better, the sample with larger treatment effects will be computed early.
Then the area of AUUC will be larger than the one via a random sorting strategy.
The result of AUUC show GALILEO gives a reasonable sorting to the RCT samples (see Fig. 14).
Finally, we search for the optimal policy via the cross-entropy method planner (Hafner et al., 2019) based on the learned model.
We test the online supply improvement in 6 cities.
The algorithm compared is a human-expert policy, which is also the behavior policy of the offline datasets.
We conduct online A/B tests for each of the cities.
For each test, we randomly split a city into two partitions, one is for deploying the optimal policy learned from the GALILEO model, and the other is as a control group, which keeps the human-expert policy as before.
Before the intervention, we collect 10 days’ observation data and compute the averaged five-minute order-taken rates as the baselines of the treatment and control group, named b t and b c respectively.
Then we start intervention and observe the five-minute order-taken rate in the following 14 days for the two groups.
The results of the treatment and control groups are y ti and y ci respectively, where i denotes the i -th day of the deployment.
The percentage points of the supply improvement are computed via difference-in-difference (DID):
[Equation]
T × 100 , where T is the total days of the intervention and T = 14 in our experiments.
[Table]
The results are summarized in Tab. 7.
The online experiment is conducted in 14 days and the results show that the policy learned with GALILEO can make better (the supply improvements are from 0.
to 1.63 percentage points) budget allocation than the behavior policies in all the testing cities .
We give detailed results which record the supply difference the the group in Fig.
[Figure]
[Table]
[Table]
0.67 ± 0.
2.04 ± 0.
SL 0.38 ± 0. 1.50 ± 0.
3.06 ± 0.
IPW 6.57 ± 1. 6.88 ± 0.
5.84 ± 0.
SCIGAN 0.74 ± 0.
2.74 ± 0.
3.19 ± 0.
t1_bias_2.
t1_bias_6. t1_bias_8.
GALILEO 0.43 ± 0.
0.25 ± 0.
0.21 ± 0. SL 0.47 ± 0.
1.33 ± 0.
1.18 ± 0. IPW 3.67 ± 2.
0.54 ± 0. 2.69 ± 1.
SCIGAN 0.45 ± 0.
1.08 ± 1.
1.01 ± 0.
t2_bias_2.
t2_bias_6. t2_bias_8.
GALILEO 1.46 ± 0.
0.85 ± 0. 0.46 ± 0.
SL 0.81 ± 0.
3.74 ± 2.
3.59 ± 0.
IPW 2.94 ± 1.
1.24 ± 0.
0.99 ± 0.
SCIGAN 0.73 ± 0.
1.20 ± 0. 2.13 ± 1.
[Table] [Table]
1.96 ± 0.
3.16 ± 0.
SL 1.92 ± 0.
2.31 ± 0.
5.11 ± 0.
IPW 7.42 ± 0.
5.36 ± 0.
5.38 ± 1.
SCIGAN 2.11 ± 0.
5.23 ± 0.
5.59 ± 1.
t1_bias_2.
t1_bias_6.
t1_bias_8.
GALILEO 1.43 ± 0.
1.09 ± 0.
1.36 ± 0.
SL 1.12 ± 0.
3.65 ± 1.
3.96 ± 1.
IPW 1.14 ± 0.
0.90 ± 0.
2.04 ± 0.
SCIGAN 3.32 ± 0.
4.74 ± 2.
5.17 ± 2.
t2_bias_2.
t2_bias_6.
t2_bias_8.
GALILEO 3.77 ± 0.
3.99 ± 0.
2.08 ± 0.
SL 2.70 ± 0.
8.33 ± 5.
9.70 ± 3.
IPW 2.92 ± 0.
3.90 ± 0.
4.47 ± 2.
SCIGAN 3.82 ± 2.
1.83 ± 1.
3.62 ± 4.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]