Original: A R EINFORCEMENT L EARNING A PPROACH TO E STIMATING L ONG - TERMT REATMENT E FFECTS 
Revised: A R EINFORCEMENT L EARNING A PPROACH TO E STIMATING L ONG - TERM E FFECTS IN N ONSTATIONARY E NVIRONMENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Randomized experiments (a.k.a.
Revised: Randomized experiments (a.k.a.

Original: A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other applications.
Revised: A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other applications.

Original: In many problems, the treatment has a lasting effect that evolves over time.
Revised: In many problems, the treatment has a lasting effect that evolves over time.

Original: A limitation with randomized experiments is that they do not easily extend to measure long-term effects, since running long experiments is time-consuming and expensive.
Revised: A limitation with randomized experiments is that they do not easily extend to measure long-term effects, since running long experiments is time-consuming and expensive.

Original: In this paper, we take a reinforcement learning (RL) approach that estimates the average reward in a Markov process.
Revised: In this paper, we take a reinforcement learning (RL) approach that estimates the average reward in a Markov process.

Original: Motivated by real-world scenarios where the observed state transition is nonstationary, we develop a new algorithm for a class of nonstationary problems, and demonstrate promising results in two synthetic datasets and one online store dataset.
Revised: Motivated by real-world scenarios where the observed state transition is nonstationary, we develop a new algorithm for a class of nonstationary problems, and demonstrate promising results in two synthetic datasets and one online store dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Randomized experiments (a.k.a.
Revised: Randomized experiments (a.k.a.

Original: A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other applications.
Revised: A/B tests) are a powerful tool for estimating treatment effects, to inform decisions making in business, healthcare and other applications.

Original: In an experiment, units like customers or patients are randomly split into a treatment bucket and a control bucket.
Revised: In an experiment, units like customers or patients are randomly split into a treatment bucket and a control bucket.

Original: For example, in a rideshare app, drivers in the control and treatment buckets are matched to customers in different ways (e.g., with different spatial ranges or different ranking functions).
Revised: For example, in a rideshare app, drivers in the control and treatment buckets are matched to customers in different ways (e.g., with different spatial ranges or different ranking functions).

Original: After we expose customers to one of these options for a period of time, usually a few days or weeks, we can record the corresponding customer engagements, and run a statistical hypothesis test on the engagement data to detect if there is a statistically significant difference in customer preference of treatment over control.
Revised: After we expose customers to one of these options for a period of time, usually a few days or weeks, we can record the corresponding customer engagements, and run a statistical hypothesis test on the engagement data to detect if there is a statistically significant difference in customer preference of treatment over control.

Original: The result will inform whether the app should launch the treatment or control.
Revised: The result will inform whether the app should launch the treatment or control.

Original: While this method has been widely successful (e.g., in online applications (Kohavi et al., 2020)), it typically measures treatment effect during the short experiment window.
Revised: While this method has been widely successful (e.g., in online applications (Kohavi et al., 2020)), it typically measures treatment effect during the short experiment window.

Original: However, in many problems, a treatment has a lasting effect that evolves over time.
Revised: However, in many problems, a treatment has a lasting effect that evolves over time.

Original: For example, a treatment that increases installation of a mobile app may result in a drop of short-term profit due to promotional benefits like discounts.
Revised: For example, a treatment that increases installation of a mobile app may result in a drop of short-term profit due to promotional benefits like discounts.

Original: But the installation allows the customer to benefit from the app, which will increase future engagements and profit in the long term.
Revised: But the installation allows the customer to benefit from the app, which will increase future engagements and profit in the long term.

Original: A limitation with standard randomized experiments is that they do not easily extend to measure long-term effects.
Revised: A limitation with standard randomized experiments is that they do not easily extend to measure long-term effects.

Original: We can run a long experiment for months or years to measure the long-term impacts, which however is time-consuming and expensive.
Revised: We can run a long experiment for months or years to measure the long-term impacts, which however is time-consuming and expensive.

Original: We can also design proxy signals that are believed to correlate with long-term engagements (Kohavi et al., 2009), but finding a reliable proxy is challenging in practice.
Revised: We can also design proxy signals that are believed to correlate with long-term engagements (Kohavi et al., 2009), but finding a reliable proxy is challenging in practice.

Original: Another solution is the surrogacy method that estimates delayed treatment impacts from surrogate changes during the experiment (Athey et al., 2019).
Revised: Another solution is the surrogacy method that estimates delayed treatment impacts from surrogate changes during the experiment (Athey et al., 2019).

Original: However, it does not estimate long-term impacts resulting from long-term treatment exposure, but rather from short-term exposure during the experiment.
Revised: However, it does not estimate long-term impacts resulting from long-term treatment exposure, but rather from short-term exposure during the experiment.

Original: In this paper, we take a reinforcement learning (RL) approach, inspired by infinite-horizon off-policy evaluation (OPE)
Revised: Shi et al. (2022b) mitigates the limitation of standard randomized experiment by framing the longterm effect as a reinforcement learning (RL) problem. Their method is closely related to recent advances in infinite-horizon off-policy evaluation (OPE)

Original: (Liu et al., 2018; Nachum et al., 2019a;
Revised: (Liu et al., 2018; Nachum et al., 2019a;

Original: Xie et al., 2019; Kallus & Uehara, 2020; Uehara et al., 2020; Chandak et al., 2021).
Revised: Xie et al., 2019; Kallus & Uehara, 2020; Uehara et al., 2020; Chandak et al., 2021).

Original: Here, the long-term treatment effect can be framed as the difference in average long-term reward of control and treatment policies. A standard RL approach uses transitions observed during a (short) experiment to predict long-term rewards.
Revised: However, their solution relies on stationary Markov assumption, which fails to capture the real-world nonstationary dynamics.

Original: Motivated by real-world scenarios where the observed state transitions are nonstationary, we consider a class of nonstationary problems, where the observation consists of two additive terms: an endogenous term that follows a stationary Markov process, and an exogenous term that is time-varying but independent of the policy.
Revised: Motivated by real-world scenarios where the observed state transitions are nonstationary, we consider a class of nonstationary problems, where the observation consists of two additive terms: an endogenous term that follows a stationary Markov process, and an exogenous term that is time-varying but independent of the policy.

Original: Based on this assumption, we develop a new algorithm to jointly estimate long-term reward and the exogenous variables.
Revised: Based on this assumption, we develop a new algorithm to jointly estimate long-term reward and the exogenous variables.

Original: Our contributions are threefold.
Revised: Our contributions are threefold.

Original: First, it is a novel application of RL to estimate long-term treatment effects, which is challenging for standard randomized experiments.
Revised: First, it is a novel application of RL to estimate long-term treatment effects, which is challenging for standard randomized experiments.

Original: Second, we develop an estimator for a class of nonstationary problems that are motivated by real-world scenarios, and give a preliminary theoretical analysis.
Revised: Second, we develop an estimator for a class of nonstationary problems that are motivated by real-world scenarios, and give a preliminary theoretical analysis.

Original: Third, we demonstrate promising results in two synthetic datasets and one online store dataset.
Revised: Third, we demonstrate promising results in two synthetic datasets and one online store dataset.

Original:  B ACKGROUND 2. L ONG -T ERM T REATMENT E FFECTS
Revised:  B ACKGROUND 2. L ONG -T ERM T REATMENT E FFECTS

Original: Let π 0 and π 1 be the control and treatment policies, used to serve individual in respective buckets.
Revised: Let π 0 and π 1 be the control and treatment policies, used to serve individual in respective buckets.

Original: In the rideshare example, a policy may decide how to match a driver to a nearby request.
Revised: In the rideshare example, a policy may decide how to match a driver to a nearby request.

Original: During the experiment, each individual (the driver) is randomly assigned to one of the policy groups, and we observe a sequence of behavior features of that individual under the influence of the assigned policy.
Revised: During the experiment, each individual (the driver) is randomly assigned to one of the policy groups, and we observe a sequence of behavior features of that individual under the influence of the assigned policy.

Original: We use variable D ∈ { 0 , 1 } to denote the random assignment of an individual to one of the policies.
Revised: We use variable D ∈ { 0 , 1 } to denote the random assignment of an individual to one of the policies.

Original: The observed features are denoted as a sequence of random variable in R d
Revised: The observed features are denoted as a sequence of random variable in R d

Original: [Equation]
Revised: [Equation]

Original: where the subscript t indicates time step in the sequence.
Revised: where the subscript t indicates time step in the sequence.

Original: A time step may be one day or one week, depending on the application.
Revised: A time step may be one day or one week, depending on the application.

Original: Feature O t consists of information like number of pickup orders.
Revised: Feature O t consists of information like number of pickup orders.

Original: We are interested in estimating the difference in average long-term reward between treatment and control policies:
Revised: We are interested in estimating the difference in average long-term reward between treatment and control policies:

Original: [Equation]
Revised: [Equation]

Original: where E averages over individuals and their stochastic sequence of engagements,  and γ ∈ (0 , 1) is the discounted factor.
Revised: where E averages over individuals and their stochastic sequence of engagements, R t = r ( O t ) is the reward signal (e.g., customer rating) at time step t , following a pre-defined reward function r : R d → R , and γ ∈ (0 , 1) is the discounted factor.

Original: The discounted factor γ is a hyper-parameter specified by the decision maker to indicate how much they value future reward over the present.
Revised: The discounted factor γ is a hyper-parameter specified by the decision maker to indicate how much they value future reward over the present.

Original: The closer γ is to 1 , the greater weight future rewards carry in the discounted sum.
Revised: The closer γ is to 1 , the greater weight future rewards carry in the discounted sum.

Original: Suppose we have run a randomized experiment with the two policies for a short period of T steps.
Revised: Suppose we have run a randomized experiment with the two policies for a short period of T steps.

Original: In the experiment, a set of n individuals are randomly split and exposed to one of the two policies π 0 and π 1 .
Revised: In the experiment, a set of n individuals are randomly split and exposed to one of the two policies π 0 and π 1 .

Original: We denote by d j ∈ { 0 , 1 } the policy assignment of individual j , and I i the index set of individuals assigned to π i , i.e., j ∈
Revised: We denote by d j ∈ { 0 , 1 } the policy assignment of individual j , and I i the index set of individuals assigned to π i , i.e., j [Equation]

Original: I i iff
Revised: I i iff

Original: d j = i .
Revised: d j = i .

Original: The in-experiment trajectory of individual j is:
Revised: The in-experiment trajectory of individual j is:

Original: [Equation]
Revised: [Equation]

Original: The in-experiment dataset is the collection of all individual data as D n = { ( τ j , d j ) } nj =1 .
Revised: The in-experiment dataset is the collection of all individual data as D n = { ( τ j , d j ) } nj =1 .

Original: Our goal is to find an estimator ˆ∆( D n ) ≈ ∆ .
Revised: Our goal is to find an estimator ˆ∆( D n ) ≈ ∆ .

Original: 2. E STIMATION UNDER S TATIONARY M ARKOVIAN D YNAMICS
Revised: 2. E STIMATION UNDER S TATIONARY M ARKOVIAN D YNAMICS

Original: Inspired by recent advances in off-policy evaluation (OPE) (e.g. Liu et al., 2018; Nachum et al., 2019b), the simplest assumption is a fully observed Markov Process that the observation in each time step can fully predict the future distribution under a stationary dynamic kernel.
Revised: Inspired by recent advances in off-policy evaluation (OPE) (e.g. Liu et al., 2018; Nachum et al., 2019b), the simplest assumption is a fully observed Markov Process that the observation in each time step can fully predict the future distribution under a stationary dynamic kernel.

Original: In this paper, we assume the dynamic kernel and reward function are both linear for simplicity, following the setting in Parr et al.
Revised: In this paper, we assume the dynamics kernel and reward function are both linear, following the setting in Parr et al.

Revised: Linear representations are popular in the RL literature (e.g., Shi et al., 2022b) , and often preferable in industrial applications due to simplicity and greater model interpretability.

Original: Assumption 2.1.
Revised: Assumption 2.1.

Original: (Linear Dynamics)
Revised: (Linear Dynamics)

Original: there is a matrix M i such that
Revised: there is a matrix M i such that

Original: [Equation]
Revised: [Equation]

Original: Remark 2.2.
Revised: Remark 2.2.

Original: Unlike standard RL, we don’t have an explicit action for a policy.
Revised: Unlike standard RL, we don’t have an explicit action for a policy.

Original: The difference between the control and treatment policy is revealed by different transition matrix M .
Revised: The difference between the control and treatment policy is revealed by different transition matrix M .

Original: Assumption 2.3.
Revised: Assumption 2.3.

Original: (Linear Reward)
Revised: (Linear Reward)

Original: There is a coefficient vector θ r ∈ R d such that
Revised: There is a coefficient vector θ r ∈ R d such that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Remark 2.4.
Revised: Remark 2.4.

Original: The reward signal may be one of the observed features.
Revised: The reward signal may be one of the observed features.

Original: For example, if we are interested in customer rating, and rating is one of the observe features, then θ r is just a one-hot vector with 1 in the corresponding coordinate.
Revised: For example, if we are interested in customer rating, and rating is one of the observe features, then θ r is just a one-hot vector with 1 in the corresponding coordinate.

Original: When the reward is complex with unknown coefficient, we can use ordinary least-squares to estimate the coefficient θ r .
Revised: When the reward is complex with unknown coefficient, we can use ordinary least-squares to estimate the coefficient θ r .

Original: Proposition 2.5.
Revised: Proposition 2.5.

Original: Under Assumption 2.1 and 2.3, if the spectral norm of M i is smaller than 1 γ , then the expected long-term reward of policy π i , v ( π i ) :
Revised: Under Assumption 2.1 and 2.3, if the spectral norm of M i is smaller than 1 γ , then the expected long-term reward of policy π i , v ( π i ) :

Original: =
Revised: =

Original: E [ (cid:80) ∞ t =

Revised: E [ (cid:80) ∞ t =

Original: γ t R t | D
Revised: γ t R t | D

Original: = i ] , can be obtained by:v [Equation] [Equation] 1 ¯ O
Revised: = i ] , can be obtained by:

Original: [Equation]
Revised: [Equation]

Original: ) 0 , where¯ O ( i ) 0 : = E [ O 0 | D = i ]
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: The only remaining step is to estimate ¯ O ( i ) 

Revised: The only remaining step is to estimate ¯ O ( i )

Original: and M i .
Revised: and M i .

Original: The former can be directly estimated from the
Revised: The former can be directly estimated from the

Original: Monte Carlo average of the experimental data: ˆ O ( i ) 

Revised: Monte Carlo average of the experimental data: ˆ O ( i )

Original: = 1 n i (cid:80) j ∈I i o 0 ,j , where n i = |I
Revised: = 1 n i (cid:80) j ∈I i o 0 ,j , where n i = |I

Original: i | is the number of individuals assigned to policy π i .
Revised: i | is the number of individuals assigned to policy π i .

Original: To estimate the latter, we may use ordinary least-squares on observed transitions:
Revised: To estimate the latter, we may use ordinary least-squares on observed transitions:

Original: [Equation]
Revised: [Equation]

Original: The detailed derivation can be found in (Parr et al., 2008).
Revised: The detailed derivation can be found in (Parr et al., 2008).

Original: Once we get the estimated value of ˆ v i ≈ v ( π i ) , the long term impact in Eq.
Revised: Once we get the estimated value of ˆ v i ≈ v ( π i ) , the long term impact in Eq.

Original: (1) can be estimated as:
Revised: (1) can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: Remark 2.6.
Revised: Remark 2.6.

Original: Although this a model-based estimator, it is equivalent to other OPE estimator in general under linear Markovian assumption (e.g., Nachum et al., 2019b; Duan et al., 2020; Miyaguchi, 2021) and it enjoys similar statistical guarantees as other OPE estimators.
Revised: Although this a model-based estimator, it is equivalent to other OPE estimator in general under linear Markovian assumption (e.g., Nachum et al., 2019b; Duan et al., 2020; Miyaguchi, 2021) and it enjoys similar statistical guarantees as other OPE estimators.

Original:  O UR M ETHOD
Revised:  O UR M ETHOD

Original: In Section 2.2, we assumed the observation
Revised: In Section 2.2, we assumed the observation

Original: O t follows a stationary Markov process, and derived a model-based closed-form solution based on linear reward Assumption 2.3.
Revised: O t follows a stationary Markov process, and derived a model-based closed-form solution based on linear reward Assumption 2.3.

Original: The weekly average metric value is highly non-stationary during holiday season.

Original: In reality, this model assumption has two major limitations.
Revised: In reality, this model assumption has two major limitations.

Original: First, real-world environments are nonstationary.
Revised: First, real-world environments are nonstationary.

Original: For example, in a hotel reservation system, seasonality heavily influences the prediction of the future booking count.
Revised: For example, in a hotel reservation system, seasonality heavily influences the prediction of the future booking count.

Original: Our stationary assumption does not capture those seasonal changes, resulting in poorly learned models and inaccurate predictions of long-term treatment effects.
Revised: Our stationary assumption does not capture those seasonal changes, resulting in poorly learned models and inaccurate predictions of long-term treatment effects.

Original: Second, in practice, we are unable to ensure that observed features fully capture the dynamics.
Revised: Second, in practice, we are unable to ensure that observed features fully capture the dynamics.

Original: OPE methods based on stationary and full observability assumptions are unlikely to work robustly in complex, real-life scenarios.
Revised: OPE methods based on stationary and full observability assumptions are unlikely to work robustly in complex, real-life scenarios.

Original: Figure 1 illustrates nonstationarity in data from an online store (see Section 5 for more details).
Revised: Figure 1 illustrates nonstationarity in data from an online store (see Section 5 for more details).

Original: The figure shows how the weekly average of a business metric changes in a span of 5 months, for two policies (C for control, and T4 for treatment).
Revised: The figure shows how the weekly average of a business metric changes in a span of 5 months, for two policies (C for control, and T4 for treatment).

Original: Such highly non-statioanary data, especially during special seasons towards the right end of the plot, are common.
Revised: Such highly non-statioanary data, especially during special seasons towards the right end of the plot, are common.

Original: However, the difference of the two policy groups remains much more stable.
Revised: However, the difference of the two policy groups remains much more stable.

Original: This is expected as both policies are affected by the same exogenous affects (seasonal variations in this example).
Revised: This is expected as both policies are affected by the same exogenous affects (seasonal variations in this example).

Original: Figure 1 motivates a relaxed model assumption (Section 3.1), by introducing a non-stationary exogenous component on top of a stationary hidden state S t .
Revised: Figure 1 motivates a relaxed model assumption (Section 3.1), by introducing a non-stationary exogenous component on top of a stationary hidden state S t .

Original: Our new assumption is that the observation O t can be decomposed additively into two parts: an endogenous part still follows a stationary
Revised: Our new assumption is that the observation O t can be decomposed additively into two parts: an endogenous part still follows a stationary

Original: Markovian dynamic for each policy group (treatment or control); and an exogenous part which is time-varying and shared across allpolicy groups.
Revised: Markovian dynamic for each policy group (treatment or control); and an exogenous part which is time-varying and shared across all groups.

Original: Based on the new assumption we propose an alternating minimization algorithm that jointly estimates both transition dynamics and exogenous variables.
Revised: Based on the new assumption we propose an alternating minimization algorithm that jointly estimates both transition dynamics and exogenous variables.

Original: 3. N ONSTATIONARY M ODEL R ELAXATION
Revised: 3. N ONSTATIONARY M ODEL R ELAXATION

Original: We assume there is an exogenous noise vector z t for each time step t , to represent the linear additive exogenous noise in the uncontrollable outside world such as seasonal effect, which applies uniformly to every individual under each treatment bucket.
Revised: We assume there is an exogenous noise vector z t for each time step t , to represent the linear additive exogenous noise in the uncontrollable outside world such as seasonal effect, which applies uniformly to every individual under each treatment bucket.

Original: We relax Assumption 2.1 as the following:
Revised: We relax Assumption 2.1 as the following:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: (Linear Additive Exogenous Noise) the observational feature O t is the sum of the endogenous hidden features and the time-varying exogenous noise z t .
Revised: (Linear Additive Exogenous Noise) the observational feature O t is the sum of the endogenous hidden features and the time-varying exogenous noise z t .

Original: [Equation]
Revised: [Equation]

Original: where z t does not depend on policy or any individual in the experiments and S t follows the linear
Revised: where z t does not depend on policy or any individual in the experiments and S t follows the linear

Original: Markovian kernel with transition matrix M i :
Revised: Markovian kernel with transition matrix M i :

Original: [Equation]
Revised: [Equation]

Original: Remark 3.2 (Explanation of the Linear Additive Model) .
Revised: Remark 3.2 (Explanation of the Linear Additive Model) .

Original: Our linear additive model is inspired by the parallel trend assumption in the Difference-in-Difference (DID) estimator (Lechner et al., 2011).
Revised: Our linear additive model is inspired by the parallel trend assumption in the Difference-in-Difference (DID) estimator (Lechner et al., 2011).

Original: In real-world environments, it is impossible to capture all the covariates that may effect the dynamics.
Revised: In real-world environments, it is impossible to capture all the covariates that may effect the dynamics.

Original: The linear additive exogenous noise z t can be seen as the drive from the outside that is both unobserved and uncontrol.
Revised: The linear additive exogenous noise z t can be seen as the drive from the outside that is both unobserved and uncontrol.

Original: For example, in an intelligent agriculture system, the highly nonstationary weather condition can be seen as exogenous which we cannot control, but the amount of water and fertilizer that affect the growth of the plant can be seen as the hidden state that is controlled by a pre-defined stationary policy.
Revised: For example, in an intelligent agriculture system, the highly nonstationary weather condition can be seen as exogenous which we cannot control, but the amount of water and fertilizer that affect the growth of the plant can be seen as the hidden state that is controlled by a pre-defined stationary policy.

Original: And we add up those two factors as the features (e.g., the condition of the crop) we observed in the real world.
Revised: And we add up those two factors as the features (e.g., the condition of the crop) we observed in the real world.

Original: From Assumption 3.1 and linear reward function assumption in 2.3, the closed form of v ( π i ) can be rewritten as:
Revised: From Assumption 3.1 and linear reward function assumption in 2.3, the closed form of v ( π i ) can be rewritten as:

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: Under Assumption 3.1 and 2.3, and suppose v ( z ∞ ) :
Revised: Under Assumption 3.1 and 2.3, and suppose v ( z ∞ ) :

Original: = (cid:80) ∞
Revised: = (cid:80) ∞

Original: t =

Revised: t =

Original: γ t z t < ∞ .
Revised: γ t z t < ∞ .

Original: Suppose the spectral norm of M i is smaller than 1 γ , the expected long-term reward can be obtained by:
Revised: Suppose the spectral norm of M i is smaller than 1 γ , the expected long-term reward can be obtained by:

Original: [Equation]
Revised: [Equation]

Original: ) , where ¯ S
Revised: , where ¯ S

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: E [ S 0 | D
Revised: E [ S [Equation] | D

Original: = i ].
Revised: = i ]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The long-term reward in Eq.
Revised: The long-term reward in Eq.

Original: (7) contains v ( z ∞ ) , which depends on the unknown exogenous noise sequence outside of the experimental window and thus is unpredictable.
Revised: (7) contains v ( z ∞ ) , which depends on the unknown exogenous noise sequence outside of the experimental window and thus is unpredictable.

Original: However, the long term treatment effect, ∆( π 1 , π 0 )
Revised: However, the long term treatment effect, ∆( π 1 , π 0 )

Original: = v ( π 1 ) − v ( π 0 ) , cancels out the dependency on that exogenous term v ( z ∞ ) .
Revised: = v ( π 1 ) − v ( π 0 ) , cancels out the dependency on that exogenous term v ( z ∞ ) .

Original: For simplicity, we redefine v ( π i ) =
Revised: For simplicity, we redefine v ( π i ) =

Original: θ ⊤ r ( I − γM i ) − 1 ¯ S
Revised: θ ⊤ r ( I − γM i ) − 1 ¯ S

Original: ( i ) 

Revised: ( i )

Original: without the term of v ( z ∞ ) .
Revised: without the term of v ( z ∞ ) .

Original: Therefore, the only thing we need to estimate is ¯ S ( i ) 

Revised: Therefore, the only thing we need to estimate is ¯ S ( i )

Original: and M i .
Revised: and M i .

Original: Once we have the access of z 0 , we can estimate ¯ S ( i ) 0 similarly as Monte Carlo sample: ˆ S 

Revised: Once we have the access of z 0 , we can estimate ¯ S ( i ) 0 similarly as Monte Carlo sample: ˆ S

Original: = 1 n i (cid:80) j ∈I i o 0 ,j − ˆ z 0 .
Revised: = 1 n i (cid:80) j ∈I i o 0 ,j − ˆ z 0 .

Original: The next question is how to estimate in-experiment exogenous variable z t and thetransition kernels of the underlying Markov Process.
Revised: The next question is how to estimate in-experiment exogenous variable z t and the underlying transition kernels.

Original: 3. O PTIMIZATION F RAMEWORK
Revised: 3. O PTIMIZATION F RAMEWORK

Original: We propose to optimize { z t } 1 ≤ t ≤ T and { M 0 , M 1 } jointly under a single loss function, with the same spirit of reducing the reconstruction loss of each transition pair similar to the model-based approach.
Revised: We propose to optimize { z t } 1 ≤ t ≤ T and { M 0 , M 1 } jointly under a single loss function, with the same spirit of reducing the reconstruction loss of each transition pair similar to the model-based approach.

Original: For each individual j in treatment group i , Assumption 3.1 implies that at time step t + 1 , the observation o j,t +1 can be written as:
Revised: For each individual j in treatment group i , Assumption 3.1 implies that at time step t + 1 , the observation o j,t +1 can be written as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] where ε j,t is a noise term with zero mean, so that M i ( o j,t − z t ) = 
Revised: where ε j,t is a noise term with zero mean, so that M i ( o j,t − z t ) = E [ S t + [Equation]

Original: E [ S t +1 | S t
Revised: S [Equation] = o [Equation]

Original: = o j,t − z t , D = i ] .
Revised: − z [Equation] D = i ] .

Original: Inspired by Eq.
Revised: Inspired by Eq.

Original: (8), given observation history D n , in order to minimize the empirical reconstruct risk by each transition pair ( o j,t , o j,t +1 ) , we construct the following loss function 
Revised: (8), given observation history D n , in order to minimize the empirical reconstruct risk by each transition pair ( o j,t , o j,t +1 ) , we construct the following loss function [Equation] To simplify the notation [Equation]

Revised: Eq.

Revised: (9) can be rewritten as a vectorized form

Original: [Equation]
Revised: [Equation]

Original: Input : In-experiment training Data D n = { ( τ j , d j ) } nj =1 , where τ i =
Revised: Input : In-experiment training Data D n = { ( τ j , d j ) } nj =1 , where τ i =

Original: ( o j, 0 , o j, 1 , . . .
Revised: ( o j, 0 , o j, 1 , . . .

Original: , o j,T ) is the in-experiment observation features for individual j , d j ∈ { 0 , 1 } is the indicator of which policy group individual j is assigned to.
Revised: , o j,T ) is the in-experiment observation features for individual j , d j ∈ { 0 , 1 } is the indicator of which policy group individual j is assigned to.

Original: Initialize the estimation of exogenous noise ˆ z = 0 .
Revised: Initialize the estimation of exogenous noise ˆ z = 0 .

Original: Optimization : while not convergent do
Revised: Optimization : while not convergent do

Original: Update M i as the ordinary least square solution given the current ˆ z : ˆ M i =  (cid:88)
Revised: Update M i as the ordinary least square solution given the current ˆ z : ˆ M i =  (cid:88)

Original: j ∈I i
Revised: j ∈I i

Original: T − 1 (cid:88) t
Revised: T − 1 (cid:88) t

Original: = ( o j,t +1 − ˆ z t +1 )
Revised: = ( o j,t +1 − ˆ z t +1 )

Original: ( o j,t − ˆ z t ) ⊤ 
Revised: ( o j,t − ˆ z t ) ⊤ 

Original:   (cid:88)
Revised:   (cid:88)

Original: j ∈I i
Revised: j ∈I i

Original: T − 1 (cid:88) t
Revised: T − 1 (cid:88) t

Original: = ( o j,t − ˆ z t )( o j,t − ˆ z t ) ⊤ 
Revised: = ( o j,t − ˆ z t )( o j,t − ˆ z t ) ⊤ 

Original:  −.
Revised:  −.

Original: Update ˆ z according to Eq.
Revised: Update ˆ z according to Eq.

Original: (12): ˆ z = ( n 0 G 0 + n 1 G 1 ) − 1 ( 1 (cid:88) i = (cid:88) j ∈I i
Revised: (12): ˆ z = ( n 0 G 0 + n 1 G 1 ) − 1 ( 1 (cid:88) i = (cid:88) j ∈I i

Original: G i o j ) .
Revised: G i o j ) .

Original: end while
Revised: end while

Original: Evaluation :
Revised: Evaluation :

Original: Compute ˆ v i =
Revised: Compute ˆ v i =

Original: θ ⊤ r ( I − γ ˆ M i ) − 1 (cid:16) ˆ O ( i ) 0 − ˆ z 0 (cid:17) , where ˆ O ( i ) 

Revised: θ ⊤ r ( I − γ ˆ M i ) − 1 (cid:16) ˆ O ( i ) 0 − ˆ z 0 (cid:17) , where ˆ O ( i )

Original: = 1 n i (cid:80) j ∈I i o 0 ,j .
Revised: = 1 n i (cid:80) j ∈I i o 0 ,j .

Original: Output the long-term impact estimation as ˆ∆ =
Revised: Output the long-term impact estimation as ˆ∆ =

Original: ˆ v 1 − ˆ v 0 .
Revised: ˆ v 1 − ˆ v 0 .

Original: To simplify the notation, Eq.

Original: (9) can be rewritten as a vectorized form

Original: L ( M 0 , M 1 , z ; D n ) = 1 (cid:88) i = (cid:88)

Original: j ∈I i ∥

Original: A i ( o j − z ) ∥ 22 , (10)

Original: where
Revised: where

Original: o j =   o j, 0 o j,. . .
Revised: o j =   o j, 0 o j,. . .

Original: o j,T   , and z =   z 0 z. . .
Revised: o j,T   , and z =   z 0 z. . .

Original: z T   are column vector aggregate over the experiment time horizon, and A i is a dT ×
Revised: z T   are column vector aggregate over the experiment time horizon, and A i is a dT ×

Original: d
Revised: d

Original: ( T + 1) matrix constructing by a block matrix M i : 
Revised: ( T + 1) matrix constructing by a block matrix M i : A i =   [Equation] [Equation]

Revised:  

Original: [Equation]
Revised: [Equation]

Original: 3. A LTERNATING M INIMIZATION
Revised: 3. A LTERNATING M INIMIZATION

Original: To reconstruct M i and z , we apply alternating minimization on the loss function L ( M 0 , M 1 , z ; D n ) in Eq.
Revised: To reconstruct M i and z [Equation] we apply alternating minimization on the loss function L ( M 0 , M 1 , z ; D n ) in Eq.

Original: By looking at the zero-gradient point of the loss function, under proper non-degenerate assumption (see Appendix for details), we have:
Revised: By looking at the zero-gradient point of the loss function, under proper non-degenerate assumption (see Appendix for details), we have:

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: Suppose ( n 0 G 0 + n 1 G 1 ) is nonsingular, the minimizer of z given M i is a closedform solution in the followings:
Revised: Suppose ( n 0 G 0 + n 1 G 1 ) is nonsingular, the minimizer of z given M i is a closedform solution in the followings:

Original: [Equation]
Revised: [Equation]

Original: where G i = A ⊤ i
Revised: i

Original: A i .
Revised: [Equation]

Original: The minimizer of M i given z is similar to Eq.
Revised: The minimizer of M i given z is similar to Eq.

Original: (5), except that we subtract the exogenous part z t from the observation:
Revised: (5), except that we subtract the exogenous part z t from the observation:

Original: [Equation]
Revised: [Equation]

Original: The final optimization process is summarized in Algorithm 1.
Revised: The final optimization process is summarized in Algorithm 1.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: We give a preliminary theoretical analysis in this section to give readers some insights on how good our estimator is once a partial oracle information is given.
Revised: We give a preliminary theoretical analysis in this section to give readers some insights on how good our estimator is once a partial oracle information is given.

Original: We will extend our analysis to quantify the error of the estimator at the convergence state of alternating minimization in future work.
Revised: We will extend our analysis to quantify the error of the estimator at the convergence state of alternating minimization in future work.

Original: To simplify our analysis, we first assume we get access to the true transition matrix M i , and our goal is to quantify the error between ˆ v ( π i ) and the true policy value v ( π i ) for each policy
Revised: To simplify our analysis, we first assume we get access to the true transition matrix M i , and our goal is to quantify the error between ˆ v ( π i ) and the true policy value v ( π i ) for each policy

Original: π i .
Revised: π i .

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: Suppose we have bounded noise and matrices under Assumption A.1 and Assumption A.2, and suppose n 0 = n 1 = n 2 is equally divided.
Revised: Suppose we have bounded noise and matrices under Assumption A.1 and Assumption A.2, and suppose n 0 = n 1 = n 2 is equally divided.

Original: When we get access of the oracle transition matrix M i
Revised: When we get access of the oracle transition matrix M i

Original: =
Revised: =

Original: M ∗ i , i ∈ { 0 , 1 } , let ˆ z =
Revised: M ∗ i , i ∈ { 0 , 1 } , let ˆ z =

Original: arg
Revised: arg

Original: min
Revised: min

Original: z L ( M ∗ 0 , M ∗ 1 , z ; D n ) .
Revised: z L ( M ∗ 0 , M ∗ 1 , z ; D n ) .

Original: If we plugin ˆ z in the estimation of ˆ v ( π i ) , we will have
Revised: If we plugin ˆ z in the estimation of ˆ v ( π i ) , we will have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: In the second analysis we assume that we get an accurate z .
Revised: In the second analysis we assume that we get an accurate z .

Original: In this case, the estimation of ˆ M reduces to the stationary assumption case in Assumption 2.1 where the hidden state variable s t =
Revised: In this case, the estimation of ˆ M reduces to the stationary assumption case in Assumption 2.1 where the hidden state variable s t =

Original: o t − z t is fully recovered.
Revised: o t − z t is fully recovered.

Original: We follow the analysis (e.g., Duan et al., 2020; Miyaguchi, 2021) of linear MDP to characterize the error.
Revised: We follow the analysis (e.g., Duan et al., 2020; Miyaguchi, 2021) of linear MDP to characterize the error.

Original: Proposition 3.6 (Proposition 11 in Miyaguchi (2021)) .
Revised: Proposition 3.6 (Proposition 11 in Miyaguchi (2021)) .

Original: Suppose we get access to the oracle exogenous noise z during the experimental period, let ˆ M i =
Revised: Suppose we get access to the oracle exogenous noise z during the experimental period, let ˆ M i =

Original: arg
Revised: arg

Original: min M i L ( { M i } , z ∗ ; D n ) in Eq.
Revised: min M i L ( { M i } , z ∗ ; D n ) in Eq.

Original: Under the assumption in Proposition 11 in Miyaguchi (2021), with the plugin estimator ˆ v with ˆ M i , we have:
Revised: Under the assumption in Proposition 11 in Miyaguchi (2021), with the plugin estimator ˆ v with ˆ M i , we have:

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: 3. P RACTICAL C ONSIDERATIONS
Revised: 3. P RACTICAL C ONSIDERATIONS

Original: Regularize the Transition Dynamic Matrices.
Revised: Regularize the Transition Dynamic Matrices.

Original: Degenerated case may happen during the alternating minimization when either 1) the spectral norm is too large, i.e. ∥ M i ∥ 2 ≥ 1 γ , leading the long-term operator ( I − γM i ) − 1 = (cid:80)
Revised: Degenerated case may happen during the alternating minimization when either 1) the spectral norm is too large, i.e. ∥ M i ∥ 2 ≥ 1 γ , leading the long-term operator ( I − γM i ) − 1 = (cid:80)

Original: ∞ t =

Revised: ∞ t =

Original: γ t M ti diverges in Eq.
Revised: γ t M ti diverges in Eq.

Original: (7), or 2) the matrix inversion calculation of M i in Eq.
Revised: (7), or 2) the matrix inversion calculation of M i in Eq.

Original: (13) is not well-defined.
Revised: (13) is not well-defined.

Original: To avoid those scenarios and stabilize the computation procedure, we add a regularization term of M i as λ i ∥
Revised: To avoid those scenarios and stabilize the computation procedure, we add a regularization term of M i as λ i ∥

Original: M i − I d ∥ 22 in our experiment.
Revised: M i − I d ∥ 22 in our experiment.

Original: The intuition is that the transition matrix should be close to identity matrix as in practice the treatment policy typically deviates from the control policy in an incremental manner.
Revised: The intuition is that the transition matrix should be close to identity matrix as in practice the treatment policy typically deviates from the control policy in an incremental manner.

Original: After adding the regularization, the closed-form minimizer of M i of the regularized loss function becomes:
Revised: After adding the regularization, the closed-form minimizer of M i of the regularized loss function becomes:

Original: [Equation]
Revised: [Equation]

Original: ⊤ 

Original:  −.

Original: Regularize the Exogenous Variable.
Revised: Regularize the Exogenous Variable.

Original: There is a challenge in deriving the closed-form z in
Revised: There is a challenge in deriving the closed-form z in

Original: Eq.
Revised: Eq.

Original: (12) where n 0 G 0 + n 1 G 1 can be degenerated or nearly degenerated.
Revised: (12) where n 0 G 0 + n 1 G 1 can be degenerated or nearly degenerated.

Original: By definition, G i is always singular.
Revised: By definition, G i is always singular.

Original: Moreover, if there is no control of the minimal eigenvalue of ( n 0 G 0 + n 1 G 1 ) , e.g. close to zero, the update step on z is uncontrolled and the variance of noise can be magnified in the direction of the minimal eigenvector.
Revised: Moreover, if there is no control of the minimal eigenvalue of ( n 0 G 0 + n 1 G 1 ) , e.g. close to zero, the update step on z is uncontrolled and the variance of noise can be magnified in the direction of the minimal eigenvector.

Original: Therefore it is crucial to regularize z .
Revised: Therefore it is crucial to regularize z .

Original: To tackle the possible degenerated circumstances, one natural idea is to include regularization of the ℓ 2 norm of z , where the regularized loss function can be written as:
Revised: To tackle the possible degenerated circumstances, one natural idea is to include regularization of the ℓ 2 norm of z , where the regularized loss function can be written as:

Original: [Equation]
Revised: [Equation]

Original: Its corresponding minimizer of ˆ z can be written as:
Revised: Its corresponding minimizer of ˆ z can be written as:

Original: [Equation]
Revised: [Equation]

Original: where I is the identity matrix of dimension d × ( T + 1) .
Revised: where I is the identity matrix of dimension d × ( T + 1) .

Original: It is worth mentioning that when the regularization parameter λ increases to infinity, z will go to 0 , and the solution reduces to the stationary case in Assumption 2.1.
Revised: It is worth mentioning that when the regularization parameter λ increases to infinity, z will go to 0 , and the solution reduces to the stationary case in Assumption 2.1.

Original: Extend to Multiple Treatment Policies
Revised: Extend to Multiple Treatment Policies

Original: The optimization framework can be easily extend to multiple treatment policies case.
Revised: The optimization framework can be easily extend to multiple treatment policies case.

Original: Suppose we have k different treatment policies π 1 , π 2 , · · · , π k and let π 0 be the control policy, the closed form solution for ˆ z under multiple dataset of different treatment groups can be derived as
Revised: Suppose we have k different treatment policies π 1 , π 2 , · · · , π k and let π 0 be the control policy, the closed form solution for ˆ z under multiple dataset of different treatment groups can be derived as

Original: [Equation]
Revised: [Equation]

Original: And the closed-form update for M i stays the same.
Revised: And the closed-form update for M i stays the same.

Original: The final estimation of the treatment effect for policy π i is ˆ∆
Revised: The final estimation of the treatment effect for policy π i is ˆ∆

Original: =
Revised: =

Original: ˆ v i − ˆ v 0 .
Revised: ˆ v i − ˆ [Equation]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Estimating long-term treatment effects  The surrogate index method (Athey et al., 2019; 2020) shares a similar goal of estimating treatment effects beyond the experiment period. They make a different assumption that the long-term effect is independent of the treatment conditioned on the surrogate index measured during the experiment.
Revised: Estimating long-term treatment effects Our work is related to causal inference with temporal data. The surrogate index method (Athey et al., 2019; 2020) makes a different assumption that the long-term effect is independent of the treatment conditioned on the surrogate index measured during the experiment.

Original: More importantly, this method does not estimate long-term impacts resulting fromlong-term treatment exposure (our focus in this work), but rather from short-term exposure during the experiment. 
Revised: It then estimates long-term impacts resulting from short-term exposure during the experiment. In contrast, our work aims to estimate long-term impacts resulting from long-term exposure. Time series methods (e.g. Bojinov & Shephard, 2019) require probabilistic treatments, which allow an individual to be exposed to different treatments at different time periods during an experiment. They then estimate the temporal treatment effect, which is averaged over all the temporal steps, differs from traditional treatment effect which is averaged over randomized individuals.

Original: Our method draws inspirations from off-policy evaluation(OPE) and related areas, whose goal is to estimate the long-term policy value, usually from a offline dataset collected under different policies.
Revised: Our method draws inspirations from off-policy evaluation(OPE) and related areas, whose goal is to estimate the long-term policy value, usually from a offline dataset collected under different policies.

Original: Most early work focuses on the family of inverse propensity score estimators that are prone to high variance in long-horizon problems (e.g., Precup et al., 2000; Murphy et al., 2001; Jiang & Li, 2016).
Revised: Most early work focuses on the family of inverse propensity score estimators that are prone to high variance in long-horizon problems (e.g., Precup et al., 2000; Murphy et al., 2001; Jiang & Li, 2016).

Original: Recently, there are growing interests in long- and even infinite-horizon settings (Liu et al., 2018;
Revised: Recently, there are growing interests in long- and even infinite-horizon settings (Liu et al., 2018;

Original: Nachum et al., 2019a; Xie et al., 2019; Tang et al., 2020; Uehara et al., 2020; Dai et al., 2020;
Revised: Nachum et al., 2019a; Xie et al., 2019; Tang et al., 2020; Uehara et al., 2020; Dai et al., 2020;

Original: Chandak et al., 2021).
Revised: Chandak et al., 2021).

Revised: In particular, Shi et al.

Revised: (2022b) considers a similar problem of estimating long-term impacts, which is comparable to our stationary baseline.

Original: These methods either rely on the stationarity assumption that is violated in many applications, or consider the general nonstationary Markov decision process (Kallus & Uehara, 2020) that does not leverage domain-specific assumptions.
Revised: However, these methods either rely on the stationarity assumption that is violated in many applications, or consider the general nonstationary Markov decision process (Kallus & Uehara, 2020) that does not leverage domainspecific assumptions.

Original: RL in nonstationary or confounded environments Our model is a special case of Partially Observable Markov Decision Process (POMDP) ( ˚Astr¨om, 1965; Kaelbling et al., 1998).
Revised: RL in nonstationary or confounded environments Our model is a special case of Partially Observable Markov Decision Process (POMDP) ( ˚Astr¨om, 1965; Kaelbling et al., 1998).

Original: OPE in general POMDPs remains challenging, unless various assumptions are made (e.g., Tennenholtz et al., 2020;
Revised: OPE in general POMDPs remains challenging, unless various assumptions are made (e.g., Tennenholtz et al., 2020;

Original: Bennett et al., 2021; Shi et al., 2022).
Revised: Bennett et al., 2021; Shi et al., 2022a).

Original: Most assumptions are on the causal relation of the logged data, such as relation between state, action and confounded variable.
Revised: Most assumptions are on the causal relation of the logged data, such as relation between state, action and confounded variable.

Original: In contrast, we make an assumption motivated by real-world data, which allows our estimator to cancel out exogenous variables from observations.
Revised: In contrast, we make an assumption motivated by real-world data, which allows our estimator to cancel out exogenous variables from observations.

Original: Our assumption is also related to MDP with Exogenous Variables (e.g., Dietterich et al., 2018; Chitnis & Lozano-P´erez, 2020), and Dynamics Parameter MDP (DPMDP) or Hidden Paramter MDP (HiP-MDP)
Revised: Our assumption is also related to MDP with Exogenous Variables (e.g., Dietterich et al., 2018; Chitnis & Lozano-P´erez, 2020), and Dynamics Parameter MDP (DPMDP) or Hidden Paramter MDP (HiP-MDP)

Original: (Al-Shedivat et al., 2017; Xie et al., 2020).
Revised: (Al-Shedivat et al., 2017; Xie et al., 2020).

Original: For exogenous variable, they assume observation features can be partitioned into two groups, where the exogenous group is not affected by the action and the endogenous group evolve as in a typical MDP.
Revised: For exogenous variable, they assume observation features can be partitioned into two groups, where the exogenous group is not affected by the action and the endogenous group evolve as in a typical MDP.

Original: The major challenge is infer the right[Figure] partition.
Revised: The major challenge is infer the right partition.

Original: Several recent works (e.g Misra et al., 2020; Du et al., 2019; Efroni et al., 2021) combine exogenous variable with rich observation in RL.
Revised: Several recent works (e.g Misra et al., 2020; Du et al., 2019; Efroni et al., 2021) combine exogenous variable with rich observation in RL.

Original: This is different from our assumption where we assume the observation is a sum of both parts, which is a more natural assumption in applications like e-commerce.
Revised: This is different from our assumption where we assume the observation is a sum of both parts, which is a more natural assumption in applications like e-commerce.

Original: For DPMDP and Hip-MDP, they assume a meta task variable which is non-stationary and changed across time but the task variable dynamic can be captured by a sequential model.
Revised: For DPMDP and Hip-MDP, they assume a meta task variable which is non-stationary and changed across time but the task variable dynamic can be captured by a sequential model.

Original: Our  assumption can be viewed as a linear special case but our focus is not to better characterize the system but is to remove the exogenous part for better predictions.
Revised: Our [Figure] assumption can be viewed as a linear special case but our focus is not to better characterize the system but is to remove the exogenous part for better predictions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our methods in three problems: a synthetic dataset, a dataset from the Type-1 Diabete
Revised: We evaluate our methods in three problems: a synthetic dataset, a dataset from the Type-1 Diabete

Original: RL simulator (Xie, 2019), and a real-world dataset from an online store.
Revised: RL simulator (Xie, 2019), and a real-world dataset from an online store.

Original: The ground truth ∆ is computed either from a true simulator or using the average of the real experimental data under a long time period.
Revised: The ground truth ∆ is computed either from a true simulator or using the average of the real experimental data under a long time period.

Original: We compare our methods based on plug-in estimator of the stationary solution in Eq.
Revised: We compare our methods based on plug-in estimator of the stationary solution in Eq.

Original: (4), its non-stationary variant in Algorithm 1, and an Naive Average baseline.
Revised: (4), its non-stationary variant in Algorithm 1, and an Naive Average baseline.

Original: The baseline directly uses the short-term reward average as the estimate of the long-term effect.
Revised: The baseline directly uses the short-term reward average as the estimate of the long-term effect.

Original: 5. S YNTHETIC S IMULATION
Revised: 5. S YNTHETIC S IMULATION

Original: The synthetic environment generates 4 randomized matrix M i for policies { π i } 3 i =

Revised: The synthetic environment generates 4 randomized matrix M i for policies { π i } 3 i =

Original: and a trajectory of randomized exogenous noise { z t } Tt =0 .
Revised: and a trajectory of randomized exogenous noise { z t } Tt =0 .

Original: See details of the synthetic dynamic in Appendix B.
Revised: See details of the synthetic dynamic in Appendix C.

Original: The randomized sequence follows the non-stationary dynamics with a parameter α controlling the scale of the exogenous noise:
Revised: The randomized sequence follows the non-stationary dynamics with a parameter α controlling the scale of the exogenous noise:

Original: o j,t = s j,t + αz t , ∀ j, t .
Revised: o j,t = s j,t + αz t , ∀ j, t .

Original: We collect n trajectories for each policy until t = T (w/ varying T ).
Revised: We collect n trajectories for each policy until t = T (w/ varying T ).

Original: We vary the parameters of the generating sequences: number n of trajectories, horizon T , data dimension d , and scale α of the exogenous noise.
Revised: We vary the parameters of the generating sequences: number n of trajectories, horizon T , data dimension d , and scale α of the exogenous noise.

Original: We plot the logarithmic Mean
Revised: We plot the logarithmic Mean

Original: Square Error (MSE) for each method in Figure 2.
Revised: Square Error (MSE) for each method in Figure 2.

Original: The result shows that our estimator method (the green line) clearly outperforms all other baselines.
Revised: The result shows that our estimator method (the green line) clearly outperforms all other baselines.

Original: Moreover, Figure 2(d) shows the increase of the scale of the exogenous noise does not affect estimation accuracy of our method.
Revised: Moreover, Figure 2(d) shows the increase of the scale of the exogenous noise does not affect estimation accuracy of our method.

Original: 5. T YPE -1 D IABETE S IMULATOR
Revised: 5. T YPE -1 D IABETE S IMULATOR

Original: This environment is modified based on an open-source implementation 1 of the FDA approved Type1 Diabetes simulator (T1DMS) (Man et al., 2014).
Revised: This environment is modified based on an open-source implementation 1 of the FDA approved Type1 Diabetes simulator (T1DMS) (Man et al., 2014).

Original: The environment simulates two-day behavior in an in-silico patient’s life.
Revised: The environment simulates two-day behavior in an in-silico patient’s life.

Original: Consumption of a meal increases the blood-glucose level in the body.
Revised: Consumption of a meal increases the blood-glucose level in the body.

Original: If the level is too high, the patient suffers from hyperglycemia.
Revised: If the level is too high, the patient suffers from hyperglycemia.

Original: If the level is too low, the patient suffers from hypoglycemia.
Revised: If the level is too low, the patient suffers from hypoglycemia.

Original: The goal is to control the blood glucose level by regulating the insulin dosage to minimize the risk associated with both hyperglycemia and hypoglycemia.
Revised: The goal is to control the blood glucose level by regulating the insulin dosage to minimize the risk associated with both hyperglycemia and hypoglycemia.

Original: We modify the Bagal and Bolus (BB) policy (Bastani, 2014) (control policy) in the codebase and set two glucose target levels and different noise levels as our two treatment policies.
Revised: We modify the Bagal and Bolus (BB) policy (Bastani, 2014) (control policy) in the codebase and set two glucose target levels and different noise levels as our two treatment policies.

Original: We collect information in the first 12-hour of all the three policies with 5000 randomized patients in each policy group and use those information to predict the long-term effect.
Revised: We collect information in the first 12-hour of all the three policies with 5000 randomized patients in each policy group and use those information to predict the long-term effect.

Original: The observation feature 
Revised: The observation feature is 2-dimensional: glucose level (CGM) and the amount of insulin injection. The non-stationarity comes from the time and amount of the consumption of a meal, which is time varying, but otherwise shared by all patients. We average a 2-day simulation window over random 250 , 000 patients as ground truth treatment effect between policy groups. [Figure]

Original: C [-G M  
-]
Revised: C [-G M  
-][Figure]

Original: {+M  
+} C G M  4000  i n s u li n 1e−   20   i n s u li n 1e−
Revised: {+M  
+} C G M  4000  i n s u li n 1e−   20   i n s u li n 1e−

Original: Stationary Nonstationary number of patients n exp horizon T number of patients n exp horizon T
Revised: Stationary Nonstationary number of patients n exp horizon T number of patients n exp horizon T

Original: [Figure]
Revised: [Figure]

Original: is 2-dimensional: glucose level (CGM) and the amount of insulin injection.

Original: The non-stationarity comes from the time and amount of the consumption of a meal, which is time varying, but otherwise shared by all patients.

Original: We average a 2-day simulation window over random 250 , 000 patients as ground truth treatment effect between policy groups.

Original: Similar to the synthetic simulator, we vary the number of patients and the experimental period.
Revised: Similar to the synthetic simulator, we vary the number of patients and the experimental period.

Original: Figure 3 shows that the non-stationary method performs better in the prediction accuracy compared to stationary method in both predictions of CGM and the amount of insulin injection.
Revised: Figure 3 shows that the non-stationary method performs better in the prediction accuracy compared to stationary method in both predictions of CGM and the amount of insulin injection.

Original: Even though the simulator is non-linear, our simple linear additive exogenous noise assumption still captures the small local changes well, which is approximately linear.
Revised: Even though the simulator is non-linear, our simple linear additive exogenous noise assumption still captures the small local changes well, which is approximately linear.

Original: 5. D ATA FROM AN O NLINE S TORE
Revised: 5. D ATA FROM AN O NLINE S TORE

Original: Metric Metric
Revised: Metric Metric

Original: Metric Metric
Revised: Metric Metric

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: We test our methods under 4 long-running experiments in an online store with a total of 7 different treatment policies (some experiments have more than 1 treatment).
Revised: We test our methods under 4 long-running experiments in an online store with a total of 7 different treatment policies (some experiments have more than 1 treatment).

Original: Each experiment has 1 control policy.
Revised: Each experiment has 1 control policy.

Original: We evaluate 4 business metrics related to customer purchases in the store (Metrics 1-4), and use d = 17 features.
Revised: We evaluate 4 business metrics related to customer purchases in the store (Metrics 1-4), and use d = 17 features.

Original: All the experiments lasted for 12 weeks.
Revised: All the experiments lasted for 12 weeks.

Original: We treat the first 5 weeks as the experiment window, and use data in those weeks to estimate long-term impacts of the 4 metrics.
Revised: We treat the first 5 weeks as the experiment window, and use data in those weeks to estimate long-term impacts of the 4 metrics.

Original: The trailing 7 -week average of the metrics are used as ground true to evaluate accuracy of various estimators.
Revised: The trailing 7 -week average of the metrics are used as ground true to evaluate accuracy of various estimators.

Original: Table 1 reports the median of the Mean Absolute Percentage Error (MAPE) of the estimators; See full results in Appendix B.
Revised: Table 1 reports the median of the Mean Absolute Percentage Error (MAPE) of the estimators; See full results in Appendix C.

Original: Given the high cost in such long-running experiments, we cannot collect more data points for comparison, and for computing statistical significance.
Revised: Given the high cost in such long-running experiments, we cannot collect more data points for comparison, and for computing statistical significance.

Original: That said, there is good evidence from the reported number that our method produces better predictions of long-term treatment effects than Naive
Revised: That said, there is good evidence from the reported number that our method produces better predictions of long-term treatment effects than Naive

Original: Average.
Revised: Average.

Original: Furthermore, our method improves on the stationary baseline, suggesting the practical relevance of our nonstationary assumtion, and effectiveness of the proposed estimator.
Revised: Furthermore, our method improves on the stationary baseline, suggesting the practical relevance of our nonstationary assumtion, and effectiveness of the proposed estimator.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we study how to estimate the long-term treatment effect by using only the inexperimental data in the non-stationary environment.
Revised: In this paper we study how to estimate the long-term treatment effect by using only the inexperimental data in the non-stationary environment.

Original: We propose a novel non-stationary RL model and an algorithm to make prediction.
Revised: We propose a novel non-stationary RL model and an algorithm to make prediction.

Original: A major limitation is the linear assumption in both the dynamics model and the additive exogenous part.
Revised: A major limitation is the linear assumption in both the dynamics model and the additive exogenous part.

Original: Once the real world model includes a highly non-linear part, the prediction value can be biased.
Revised: Once the real world model includes a highly non-linear part, the prediction value can be biased.

Original: Future direction includes further relax our model to nonlinear case to better capture the real world environment.
Revised: Future direction includes further relax our model to nonlinear case to better capture the real world environment.

Original: Appendix A P ROOF
Revised: Appendix A P ROOF

Original: In this section, we provide detailed proof for the theorem in the main text, as a self-contained section, we briefly introduce the notation as below, and adopt the regularized, multiple policy groups settings in the appendix:
Revised: In this section, we provide detailed proof for the theorem in the main text, as a self-contained section, we briefly introduce the notation as below, and adopt the regularized, multiple policy groups settings in the appendix:

Original: • n : number of total individuals.
Revised: • n : number of total individuals.

Original: • I i : the index set for policy π i ; n i = |I
Revised: • I i : the index set for policy π i ; n i = |I

Original: i | as the number of individual in under policy π i .
Revised: i | as the number of individual in under policy π i .

Original: • k total number of different policy group.
Revised: • k total number of different policy group.

Original: • D n : dataset for n individuals in the experimental period.
Revised: • D n : dataset for n individuals in the experimental period.

Original: In the appendix, we denote the ground truth dynamic M ∗ i and the ground truth exogenous noise z ∗ with a star ∗ to distinguish the variables M i and z during optmization process.
Revised: In the appendix, we denote the ground truth dynamic M ∗ i and the ground truth exogenous noise z ∗ with a star ∗ to distinguish the variables M i and z during optmization process.

Original: A. A SSUMPTIONS
Revised: A. A SSUMPTIONS

Original: The dynamic assumption of our linear additive exogenous noise assumption in Assumption 3.1 can be rewritten as the following equation:
Revised: The dynamic assumption of our linear additive exogenous noise assumption in Assumption 3.1 can be rewritten as the following equation:

Original: [Equation]
Revised: [Equation]

Original: (15) where ε j,t is a zero-mean noise.
Revised: (15) where ε j,t is a zero-mean noise.

Original: Let ε j = 
Revised: Let ε j = 

Original:  ε j
Revised:  ε j

Original: [Equation]
Revised: [Equation]

Original: ε j,. . .
Revised: ε j,. . .

Original: [Equation]
Revised: [Equation]

Original: forms a martingale
Revised: forms a martingale

Original: [Equation]
Revised: [Equation]

Original: (16) where the filtration F j
Revised: (16) where the filtration F j

Original: =
Revised: =

Original: o 1 , ..., o j − 1 } is the information up to the first j − 1 individuals.
Revised: o 1 , ..., o j − 1 } is the information up to the first j − 1 individuals.

Original: We make addition bounded assumption on the zero-mean noise term for the proof:
Revised: We make addition bounded assumption on the zero-mean noise term for the proof:

Original: Assumption A.1 (Bounded Noise assumption) .
Revised: Assumption A.1 (Bounded Noise assumption) .

Original: Let ε
Revised: Let ε

Original: j
Revised: j

Original: = M ∗ i ( o j,t − z ∗ t ) − ( o j,t +1 − z ∗ t +1 ) , j ∈
Revised: = M ∗ i ( o j,t − z ∗ t ) − ( o j,t +1 − z ∗ t +1 ) , j ∈

Original: I i be the residual of the transition under the true transition matrix M ∗ i , we have
Revised: I i be the residual of the transition under the true transition matrix M ∗ i , we have

Original: [Equation]
Revised: [Equation]

Original: (17) where C ε is a uniform constant independent of policy assignment.
Revised: (17) where C ε is a uniform constant independent of policy assignment.

Original: For the empirical covariance matrix in the middle step of the calculation, we assume they are all bounded.
Revised: For the empirical covariance matrix in the middle step of the calculation, we assume they are all bounded.

Original: Assumption A.2 (Bounded Norm for Matrices) .
Revised: Assumption A.2 (Bounded Norm for Matrices) .

Original: We make the following assumptions on matrices
Revised: We make the following assumptions on matrices

Original: ∥
Revised: ∥

Original: M ∗ i ∥
Revised: M ∗ i ∥

Original: ≤ C M i < 1 γ , ∀ i .
Revised: ≤ C M i < 1 γ , ∀ i .

Original: ∥ (Λ ∗ n /n ) − 1 ∥ ≤ C Λ .
Revised: ∥ (Λ ∗ n /n ) − 1 ∥ ≤ C Λ .

Original: A. L OSS FUNCTION AND A LTERNATING M INIMIZATION
Revised: A. L OSS FUNCTION AND A LTERNATING M INIMIZATION

Original: Our loss function can be written as:
Revised: Our loss function can be written as:

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Fix { M i } , denote G i =
Revised: Fix { M i } , denote G i =

Original: A ⊤ i
Revised: A ⊤ i

Original: A i where A i is defined in Eq.
Revised: A i where A i is defined in Eq.

Original: (11) , the minimization of z = arg min z L ( { M i } 1 ≤ i ≤ k , z ; D n ) is
Revised: (11) , the minimization of z = arg min z L ( { M i } 1 ≤ i ≤ k , z ; D n ) is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By taking the gradient of the loss function, we will have:
Revised: By taking the gradient of the loss function, we will have:

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Here, G i =
Revised: Here, G i =

Original: A ⊤ i
Revised: A ⊤ i

Original: A i is semi-definite, so the inversion of the large matrix in the right side of the expression always exists.
Revised: A i is semi-definite, so the inversion of the large matrix in the right side of the expression always exists.

Original: Similarly we can get the minimizer of M i fixing z .
Revised: Similarly we can get the minimizer of M i fixing z .

Original: Lemma A.4.
Revised: Lemma A.4.

Original: By fixing z , the minimizer of M i can be written as
Revised: By fixing z , the minimizer of M i can be written as

Original: M i ( z
Revised: M i ( z

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is similarly applied by looking at the zero gradient of M i .
Revised: The proof is similarly applied by looking at the zero gradient of M i .

Original: If we set λ i = 0 and z = 0 , the minimization reduces back to estimation of ˆ M in Eq.
Revised: If we set λ i = 0 and z = 0 , the minimization reduces back to estimation of ˆ M in Eq.

Original: A. E RROR A NALYSIS
Revised: A. E RROR A NALYSIS

Original: Lemma A.5.
Revised: Lemma A.5.

Original: Let M ∗
Revised: Let M ∗

Original: i be the true dynamic of the underlying state, we have:
Revised: i be the true dynamic of the underlying state, we have:

Original: [Equation]
Revised: [Equation]

Original: where Λ ∗ n = λ z I d × T +
Revised: where Λ ∗ n = λ z I d × T +

Original: [Equation]
Revised: [Equation]

Original: ki =1 n i G ∗
Revised: ki =1 n i G ∗

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By expand the definition of z ( { M ∗ i }
Revised: By expand the definition of z ( { M ∗ i }

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma A.6.
Revised: Lemma A.6.

Original: Let z ∗ be the true exogenous noise, we have:
Revised: Let z ∗ be the true exogenous noise, we have:

Original: [Equation]
Revised: [Equation]

Original: where Σ ∗ n = (cid:80) j ∈I i (cid:80) T − 1 t =1 ( o j,t − z ∗ t )(
Revised: where Σ ∗ n = (cid:80) j ∈I i (cid:80) T − 1 t =1 ( o j,t − z ∗ t )(

Original: o j,t − z ∗ t ) ⊤ is the empirical covariace matrix.
Revised: o j,t − z ∗ t ) ⊤ is the empirical covariace matrix.

Original: Proof.
Revised: Proof.

Original: By expand the definition of M i ( z ∗ )
Revised: By expand the definition of M i ( z ∗ )

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION 2. Proof
Revised: A. P ROOF OF P ROPOSITION 2. Proof

Original: By induction, it is not hard to prove that
Revised: By induction, it is not hard to prove that

Original: [Equation]
Revised: [Equation]

Original: Sum up all condition on O 0 , we have:
Revised: Sum up all condition on O 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: By the definition of long-term discounted reward G , we have:
Revised: By the definition of long-term discounted reward G , we have:

Original: [Equation]
Revised: [Equation]

Original: where the last equation holds when ∥ M i ∥ < 1 γ .
Revised: where the last equation holds when ∥ M i ∥ < 1 γ .

Original: A. P ROOF OF P ROPOSITION 3. 
Revised: A. P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: From Lemma A.5, suppose λ z = 0 and (Λ ∗ n ) − 1 exists, we have:
Revised: From Lemma A.5, suppose λ z = 0 and (Λ ∗ n ) − 1 exists, we have:

Original: [Equation]
Revised: [Equation]

Original: Consider ˆ v ( π 0 ) if we plugin ˆ z and the true dynamic M ∗ 0 , the error between ˆ v and v is
Revised: Consider ˆ v ( π 0 ) if we plugin ˆ z and the true dynamic M ∗ 0 , the error between ˆ v and v is

Original: [Equation]
Revised: [Equation]

Original: where β r =
Revised: where β r =

Original: ( I − γM ∗ 0 ) − T θ r , and ˜ β r is the extended vector of β r if we fill the other vector value at other time step as 0 .
Revised: ( I − γM ∗ 0 ) − T θ r , and ˜ β r is the extended vector of β r if we fill the other vector value at other time step as 0 .

Original: Expand the difference ( z 0 − z ∗ 0 ) we have:
Revised: Expand the difference ( z 0 − z ∗ 0 ) we have:

Original: [Equation]
Revised: [Equation]

Original: By Assumption A.1 and Assumption A.2, the norm of ˜ β r is the same as β r , which is bounded by ∥ β r ∥ ≤ 
Revised: By Assumption A.1 and Assumption A.2, the norm of ˜ β r is the same as β r , which is bounded by ∥ β r ∥ ≤ [Equation] θ

Original: [Equation]
Revised: [Equation]

Revised: ∥ .

Original: The matrix norm in the middle  is bounded because of Assumption A.2.
Revised: The matrix norm in the middle factor is bounded because of Assumption A.2.

Original: And by vector concentration inequality, since ε j is norm-subGaussian(Jin et al., 2019), there exist a constant c that with probability at least 1 − δ :
Revised: Finally, by vector concentration inequality, since ε j is norm-subGaussian (Jin et al., 2019), there exist a constant c that with probability at least 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: In sum, the error is bounded by O ( 1 √ n ) with probability at least 1 − δ , and the constant is related to
Revised: In sum, the error is bounded by O ( 1 √ n ) with probability at least 1 − δ , and the constant depends on

Original: C ε , C M i , C Λ and the norm of ∥
Revised: C ε , C M i , C Λ and the norm of ∥

Original: θ r ∥ .
Revised: θ r ∥ .

Original: A. P ROOF OF P ROPOSITION 3. 
Revised: A. P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: Since we get access to the ground true z ∗ , the remaining problem is by changing the state as s j,t =
Revised: Since we get access to the ground true z ∗ , the remaining problem is by changing the state as s j,t =

Original: o j,t − z ∗ t and reduce the problem back to standard MDP.
Revised: o j,t − z ∗ t and reduce the problem back to standard MDP.

Original: The detailed proof can refer to
Revised: The detailed proof can refer to

Original: Proposition 11 in Miyaguchi (2021).
Revised: Proposition 11 in Miyaguchi (2021).

Original: B E XPERIMENTS D ETAILS B. S YNTHETIC S IMULATION
Revised: B R EDUCE THE C OMPUTATION C OMPLEXITY WITH P RE - COMPUTATION

Revised: In this section, we explain how to reduce the computation complexity with pre-computation.

Revised: Pre-computation.

Revised: Compute

Revised: [Equation]

Revised:   (cid:88)

Revised: j ∈I i

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: 

Revised:  −and

Revised: ¯ o t = (cid:88)

Revised: j ∈I i

Revised: o j,t .

Revised: The pre-computation requires computation complexity of O ( nTd 2 + d 3 ) , where d 2 is the computation complexity of the outer product, d 3 is the computation complexity of the matrix inversion after summing up the matrix.

Revised: In Each Iteration.

Revised: The computation of M can be rewritten as

Revised: [Equation]

Revised:  (cid:88)

Revised: [Equation]

Revised: 

Revised:  

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: which requires computation complexity of O ( Td 2 ) .

Revised: Similarly, the computation of z ( G ) =

Revised: ( k (cid:88) i

Revised: = n i G i ) − 1 ( k (cid:88) i =

Revised: G i ¯ o j ) requires computation complexity of O ( T 2 d 2 ) .

Revised: Both steps are computationally scalable, since it does not rely on number of individuals n (which is often much larger than T and d ).

Revised: Overall Computation Complexity.

Revised: Suppose we execute the iterations for k times, then the total computation complexity for the alternating minimization is O ( nTd 2 + d 3 + kT 2 d 2 ) .

Revised: In practice, the number of different individual n is far larger than the experiment horizon T and the feature dimension d , therefore the computation complexity essentially scales linearly with n .

Revised: C E XPERIMENTS D ETAILS C. S YNTHETIC S IMULATION

Original: The synthetic environment generates 4 randomized matrix M i for policies { π i } 3 i =0 , where each entry of M i is a positive number randomly sample from a uniform distribution between (0 , 1) .
Revised: The synthetic environment generates 4 randomized matrix M i for policies { π i } 3 i =0 , where each entry of M i is a positive number randomly sample from a uniform distribution between (0 , 1) .

Original: We normalize each row so that it sums up to 1 , and we set ˜ M i = 0 .
Revised: We normalize each row so that it sums up to 1 , and we set ˜ M i = 0 .

Original: 5 I + 0 .
Revised: 5 I + 0 .

Original: 5 M i as our final transition matrix.
Revised: 5 M i as our final transition matrix.

Original: The 0 .
Revised: The 0 .

Original: 5 I part ensures each matrix is not too far away from each other.
Revised: 5 I part ensures each matrix is not too far away from each other.

Original: We generate a set of i.i.d.
Revised: We generate a set of i.i.d.

Original: random vector η t ∼ N (0 , 1 . 5 I ) and set z t
Revised: random vector η t ∼ N (0 , 1 . 5 I ) and set z t

Original: +1 =
Revised: +1 =

Original: z t + η t recursively.
Revised: z t + η t recursively.

Original: And we let ˜ z t = α t ∗ z t as the final exogenous noise, where α t = e β t and β t ∼ N (0 , 0 .
Revised: And we let ˜ z t = α t ∗ z t as the final exogenous noise, where α t = e β t and β t ∼ N (0 , 0 .

Original: 5 I ) , i.i.d. .
Revised: 5 I ) , i.i.d. .

Original: All the parameters ( z t and M i ) of the dynamic are fixed once generated, and we use the dynamic to generate our observation for each individual, following
Revised: All the parameters ( z t and M i ) of the dynamic are fixed once generated, and we use the dynamic to generate our observation for each individual, following

Original: [Equation]
Revised: [Equation]

Original: where ε t is independently drawn from a standard normal distribution, and α control the level of exogenous noise.
Revised: where ε t is independently drawn from a standard normal distribution, and α control the level of exogenous noise.

Original: B. P OLICY CONSTRUCTION IN T YPE -1 D IABETE S IMULATOR
Revised: C. P OLICY CONSTRUCTION IN T YPE -1 D IABETE S IMULATOR

Original: The Bagal and Bolus policy is a parametrized policy based on the amount of insulin that a person with diabetes is instructed to inject prior to eating a meal (Bastani, 2014)injection = current blood glucose [Equation] target blood glucose
Revised: The Bagal and Bolus policy is a parametrized policy based on the amount of insulin that a person with diabetes is instructed to inject prior to eating a meal (Bastani, 2014)

Original: [Equation]
Revised: [Equation]

Original: where CF and CR are parameter based on patients information such as body weights, which is already specified in the simulator.
Revised: where CF and CR are parameter based on patients information such as body weights, which is already specified in the simulator.

Original: We set our two treatment policies with target blood glucose level at 145 and 130 (compared to control: 140 ).
Revised: We set our two treatment policies with target blood glucose level at 145 and 130 (compared to control: 140 ).

Original: And we increase the noise in the insulin pump simulator in both the treatment policies.
Revised: And we increase the noise in the insulin pump simulator in both the treatment policies.

Original: B. F ULL RESULTS FOR ALL THE ONLINE STORE EXPERIMENTS .
Revised: C. R ANDOM P ATIENTS G ENERATION IN T YPE -1 D IABETE S IMULATOR

Revised: Type-1 Diabete simulator pre-stores 30 patients parameter. To randomly generate a new patient, we randomly pick two different patients A and B , and use a random linear coefficient β ∼ U (0 , 0 . 2) and mixed the parameter of a new patient as

Original: [Equation]
Revised: [Equation]

Revised: where θ A and θ B are the parameters of patients A and B, respectively.

Original: Naive Average 122.47% 93.61% 51.20%
Revised: Since patient A has more weight of the parameter, the parameters in Bagal and Bolus policy, CF and CR , follow patient A ’s parameter. C. F ULL RESULTS FOR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]