P ERFORMANCE B OUNDS FOR M ODEL AND P OLICY T RANSFER IN H IDDEN - PARAMETER MDP S
Anonymous authors
Paper under double-blind review
A BSTRACT
In the Hidden-Parameter MDP (HiP-MDP) framework, a family of reinforcement learning tasks is generated by varying hidden parameters specifying the dynamics and reward function for each individual task.
The HiP-MDP is a natural model for families of tasks in which meta- and lifelong-reinforcement learning approaches can succeed.
Given a learned context encoder that infers the hidden parameters from previous experience, most existing algorithms fall into two categories: model transfer and policy transfer , depending on which function the hidden parameters are used to parameterize.
We characterize the robustness of model and policy transfer algorithms with respect to hidden parameter estimation error.
We first show that the value function of HiP-MDPs is Lipschitz continuous under certain conditions.
We then derive regret bounds for both settings through the lens of
Lipschitz continuity.
Finally, we empirically corroborate our theoretical analysis by varying the hyper-parameters governing the Lipschitz constants of two continuous control problems; the resulting performance is consistent with our theories.
I NTRODUCTION
[Equation]
[Figure]
Hidden-parameter Markov Decision Processes (HiP-MDPs)
(Doshi-Velez & Konidaris, 2016) describe a family of related tasks by modeling task variations with a set of low-dimensional hidden parameters.
HiP-MDPs are widely used in recent metaReinforcement Learning (RL) and lifelong RL works, in which an agent needs to quickly adapt to new tasks by transferring the knowledge from previous tasks.
To solve HiP-MDPs, most existing algorithms first infer the hidden parameters from previous experience (e.g. by learning a context encoder that maps trajectories to a hidden parameter estimate θ ), then use the estimated hidden parameters to solve new tasks with either model (Killian et al., 2017; Lee et al., 2020;
Fu et al., 2021b) or policy transfer (Yao et al., 2018;
Rakelly et al., 2019) algorithms.
As shown in Figure 1, in model transfer, the inferred hidden parameters are used to help build a simulator of the environment, which then can be used for planning.
In policy transfer, the inferred hidden parameters are used to parameterize the policy of the new task directly.
Previous works have observed mixed evidence of when model/policy transfer can be better than the other empirically (Yao et al., 2018; Lee et al., 2020; Fu et al., 2021b).
In this work, we take a step in the theoretical direction by studying the regret bounds of model and policy transfer algorithms, respectively, which helps characterize and understand the robustness of these algorithms.
Two main factors affect the performance of HiP-MDP algorithms: (1) the estimation accuracy of the hidden parameter, which will be used as an additional input for either the learned model / policy, and (2) the quality of the learned model / policy (i.e., given an accurately estimated hidden parameter, whether the agent can reach the optimal performance).
In practice, the HiP-MDP tasks considered in most of the recent meta-RL papers (e.g., changing the physical properties of MuJoCo-simulated robots) are relatively easy to solve when the hidden parameters are given.
We therefore assume the learned model and policy are optimal and theoretically analyze how hidden parameter estimation error affects the final performance of model and policy transfer algorithms.
Our contributions are as follows.
We first derive the conditions under which the value function of
HiP-MDPs is Lipschitz continuous.
We then derive the upper bounds of regret for model and policy transfer algorithms, respectively, when the estimation error of the hidden parameters is bounded.
We also give an upper bound for multi-step prediction error in model transfer.
We further prove that the derived bounds are tight when the transition dynamics, reward function and policy are linear and deterministic.
Given the same hidden parameter estimation error bound, our results characterize when model/policy transfer can be more robust than the other: a slower increase in the Lipschitz constant with respect to the hidden parameter implies more robustness.
In addition to theoretical analysis, we empirically study the performance of model and policy transfer algorithms in two continuous control domains.
For each domain, we control the Lipschitz constants of HiP-MDPs by altering the hyper-parameters of the environments.
The results are consistent with our theoretical understanding about how the regrets of model and policy transfer algorithms scale with respect to the estimation error of the hidden parameters: A slower increase in Lipschitz constant with respect to the hidden parameter implies a smaller performance decay.
B ACKGROUND AND R ELATED W ORK
HiP-MDPs model the variations in the transition dynamics and reward functions by assigning each task a hidden parameter θ , drawn from the distribution P Ω .
The agent neither observes θ nor has access to the the distribution P Ω that generates the task family.
For a given task, parameterized by θ ∈ Θ , the stochastic dynamics are given by T ( s ′ | s, a ; θ ) and the deterministic reward function by R ( s, a ; θ ) .
We consider continuous state and action space in this work ( s ∈ S, a ∈ A ) .
Upon encountering a new task, the agent estimates the new dynamics, ˆ T , and the new reward function, ˆ R , by inferring a distribution ˆ ω ( θ ) over the hidden parameter.
If our estimation of θ is accurate, then ˆ ω ( θ ) should be close to the true distribution ω ( θ ) , i.e. peaking at the true θ for this specific task.
We assume the learned dependency of ˆ T and ˆ R on θ is accurate.
Thus, if the estimation of θ is also accurate, the agent can solve the task completely.
A number of approaches have been proposed to infer the hidden parameters of HiP-MDPs in the latent representation space, such as using Bayesian models to leverage prior knowledge (Killian et al., 2017; Yao et al., 2018; Fu et al., 2021b), or training a context encoder that maps trajectories to the latent parameter (Rakelly et al., 2019; Zintgraf et al., 2020; Lee et al., 2020).
Given the inferred hidden parameters, various RL algorithms have been used to adapt to the new tasks.
The downstream RL algorithm given the inference of the hidden parameter includes training an off-policy actor-critic that takes in the estimated parameter as an additional input (Rakelly et al., 2019; Fakoor et al., 2020; Fu et al., 2021a), or leveraging recurrent neural networks (Zintgraf et al., 2020; Duan et al., 2016), or planning based on the learned transition model that also takes in the inferred parameter (Lee et al., 2020; Mendonca et al., 2020).
Yet, in practice, there is no clear winner among these algorithms, leaving the question that what factors affect the performance of different algorithms, and how.
Nair & Doshi-Velez study a similar problem but in a different setting where the hidden parameters are given (contextual MDP) and derive sample complexity bounds for model-based learning.
In this work, we quantify how errors and uncertainty in estimating θ affect the performance of model and policy transfer methods through the lens of Lipschitz continuity.
Given a distance metric on the space M , d M , Lipschitz continuity quantifies the smoothness of a function as follows.
Definition 2.1.
A function f : M 1 (cid:55)→ M 2 is uniformly Lipschitz continuous if
[Equation]
We use K M 1 f in some places for simplicity of notation.
Throughout the work, we use the Euclidean distance for d in the case of point vectors and the Wasserstein distance in the case of probability distributions.
The main goal of this work can now be stated as deriving the relation between d ( ω ( θ ) , ˆ ω ( θ )) ≤ ϵ , and the performance of model transfer and policy transfer methods.
The Wasserstein Metric can be linked to Lipschitz continuity using duality (Villani, 2008):
[Equation]
Given a distribution ω ( θ ) over the hidden parameters, we define the generalized transition, reward and value functions in HiP-MDPs as:
[Equation]
These generalised functions are an expectation over the uncertainty in the hidden parameter.
All the
Q and V functions we use below refer to the generalized function Q g and V g .
We are now equipped to formally state all the smoothness assumptions we make about the HiP-MDPs:
Definition 2.2.
A HiP-MDP is ( K SR , K ST , K AR , K AT , K Ω R , K Ω T ) -Lipschitz
if, for all ω ∈ Ω , and s ∈ S and a ∈ A :
[Equation]
To simplify notation, we will write that a HiP-MDP is Lipschitz continuous when all the constants can be implied from the context.
In the following theoretical analysis, we assume the HiP-MDPs are Lipschitz.
Intuitively, this implies that when the hidden parameters of two MDPs are close, their transition dynamics and reward functions are similar, and for a given task, transitions dynamics and rewards do not change much given small changes in the states and actions.
Similar assumptions have commonly been made in previous works (Hinderer, 2005; Asadi et al., 2018; Gottesman et al., 2021; Gelada et al., 2019;
Ren et al., 2019;
Luo et al., 2019).
We extend the smoothness assumption to the hidden parameters for a better understanding about the role of hidden parameter estimation error in the final performance.
In addition to the smoothness assumptions about the HiP-MDPs, we make the following assumptions about the smoothness of policies.
Definition 2.3.
A policy π ( a | s, ω ( θ )) is ( K Sπ , K Ω π ) -Lipschitz continuous if:
[Equation]
are finite.
We evaluate policies by calculating their expected discounted reward given a distribution of initial states, µ ( s ) :
[Equation]
Lipschitz continuity in MDPs has been studied by many previous works (Hinderer, 2005; Rachelson & Lagoudakis, 2010;
Tang et al., 2020; Gottesman et al., 2021).
Pirotta et al.
(2015) propose to use
Lipschitz continuity property of MDPs to speed up policy gradient algorithms.
Asadi et al.
(2018) show how the magnitude of the Lipschitz constant of the transition dynamics affects the performance of model-based RL algorithm’s performance is influenced by the magnitude of the Lipschitz constant of the model.
Gelada et al.
(2019) leverages Lipschitz continuity assumptions in regular MDPs to learn state abstractions.
However, none of them conduct theoretical analysis on HiP-MDPs.
L IPSCHITZ VALUE FUNCTION OF H I P-MDP S
Our first main result is showing under which conditions the value functions are Lipschitz continuous for HiP-MDPs.
We show this through a combination of Theorem 3.1 and Theorem 4.3 & 5.2 in the following sections.
Theorem 3.1 mainly studies the Lipschitz continuity with respect to S and A while the hidden parameter is fixed.
This can be seen as a generalization of theorems derived in prior papers (Rachelson & Lagoudakis, 2010;
Pirotta et al., 2015; Asadi et al., 2018; Gelada et al., 2019) that study Lipschitz value function for a regular MDP.
Theorem 3.1.
For a ( K SR , K ST , K AR , K AT , K Ω R , K Ω T ) -Lipschitz HiP-MDP with a ( K Sπ , K Ω π ) -Lipschitz policy π
[Equation]
if γ ( K ST + K Sπ K AT ) < 1 , 1 the value function is Lipschitz continuous with respect to S and
A with constants bounded by:
[Equation]
Proof.
(sketch)
We derive the value difference bound by decomposing it into the combination of the Q-value difference with respect to S & A using our previous definitions and the Bellman update rule (9).
Then we get the bounds for K AQ π and K SQ π respectively by applying the dual form of the
Wasserstein Metric (Equation 2) and with the fixed-point iteration.
See appendix E for all proofs.
The above theorem shows the conditions under which the value functions of HiP-MDPs are Lipschitz continuous with respect to states and actions.
We can get the following corollary when the policy is optimal:
Corollary 3.2.
If the policy π is optimal and γK ST < 1 , then:
[Equation]
Note that in contrast to K SQ ∗ , K AQ ∗ is affected by the smoothness of the transition dynamics and reward functions with respect to action space ( K AT and K AR ).
Theorem 3.1 and Corollary 3.2 also directly apply to a ( K SR , K ST , K AR , K AT ) -Lipschitz regular MDP with a K Sπ -Lipschitz policy, as the Lipschitz continuity assumptions with respect to hidden parameters are not used in both the final results and proofs.
Specifically, the upper bound for K SQ ∗ in Corollary 3.
recovers the results in (Asadi et al., 2018; Gelada et al., 2019).
If we assume the Lipschitz constants for state and action are the same, we get Corollary E.1, which recovers the results in (Rachelson &
Lagoudakis, 2010; Pirotta et al., 2015).
For regular MDPs, our results are more general compared to prior works as (1) we do not assume that the Lipschitz constants for state and action are the same because this is usually not the case in practice; (2) our theory applies to all Lipschitz policies rather than only the optimal policy.
One of our major goals in this work is to derive the conditions for Lipschitz value functions of
HiP-MDPs.
Theorem 3.1 shows that the value functions in HiP-MDPs are Lipschitz continuous with respect to S and A under certain conditions.
To complete the theorem for Lipschitz value function in HiP-MDPs, we also need to explore the conditions under which the value function is Lipschitz continuous with respect to the remaining input of Q-function in HiP-MDPs, ω .
As we shown in the next two sections, exploring this actually results in different bounds for policy transfer and model transfer.
By leveraging the results of Lipschitz value functions, we then find how the estimation error d ( ω ( θ ) , ˆ ω ( θ )) ≤ ϵ of the hidden parameter at test time will affect the performance of model and policy transfer algorithms respectively.
We quantify the performance gap by deriving the upper bound of the regret as a function of the error in estimating the hidden parameter.
Note that throughout the following analysis, we abstract out the discussion of how ˆ ω ( θ ) is estimated, and assume it is given with an estimation error bounded by ϵ .
P ERFORMANCE BOUNDS OF M ODEL T RANSFER IN H I P-MDP S
We now focus on the regret of model transfer in HiP-MDPs induced by the hidden parameter estimation error, d ( ω ( θ ) , ˆ ω ( θ )) ≤ ϵ .
We assume that the agent has learnt an accurate transition function ˆ T g ( s ′ | s, a, ω ( θ )) and reward function ˆ R g
( s, a, ω ( θ )) .
Upon encountering a new task, the agent first estimates the hidden parameter and then feeds it into the learned transition function ˆ T g and reward function ˆ R g .
The agent will use them to generate samples and train a task-specific policy.
The algorithm framework can be found in Algorithm 2 and Figure 1.
Note that a similar paradigm has been widely used (Lee et al., 2020; Mendonca et al., 2020) in complex continuous control tasks.
We first derive a bound of the compounding error of dynamics prediction given the estimation error ϵ in hidden parameter at test time, with the Lipschitz model class (Asadi et al., 2018)
assumption (we include a definition in appendix E):
Theorem 4.1.
Assume the estimation error for hidden parameter θ is bounded by ϵ , that is,
W ( ω ( θ ) , ˆ ω ( θ )) ≤ ϵ .
Further assume an accurately learned ˆ T g induced by a Lipschitz model class F g with the Lipschitz constant K Ωˆ T and K S ˆ T , a fixed sequence of actions a 0 , · · · , a n − 1 , and an initial state distribution µ ( s ) .
Then ∀ n ≥ 1 :
[Equation]
where ˆ T ng ( ·| µ, ω )) :
=
ˆ T g ( ·| ˆ T g ( ·| ...
ˆ T g ( ·| µ, a 0 , ω ) ..., a n −
[Equation]
ω )
[Equation]
a n − 1 , ω ) , and ˆ T ng ( ·| µ, ˆ ω )) is defined similarly.
The result shows how the multi-step prediction error ξ of a generalized transition function in HiPMDP scales with respect to the hidden parameter estimation error ϵ .
Additionally, the smoothness of the transition function also affects the multi-step prediction error.
As we show in the experiments, this multi-step prediction bound can have a huge impact on the planning performance for model transfer.
Then, we investigate the value estimation difference and regret of model transfer induced by the hidden parameter estimation error.
Note that the remaining theoretical results in this paper do not involve the Lipschitz model class assumption.
Lemma 4.2.
Given a HiP-MDP with learned ( K Ωˆ R , K Ωˆ T ) -Lipschitz transition model ˆ T , reward function ˆ R , in model transfer , the generalized value function is Lipschitz continuous with respect to Ω with a constant bounded by:
[Equation]
We get the above bound for the Lipschitz value function when assuming the Lipschitz continuity of transition and reward function with respect to only the hidden parameter.
The lemma shows the relationship between K Ω V π and K Ω Q π in model transfer case.
Then, By further assuming the Lipschitz continuity of dynamics & policy in Lemma 9, we can get the following bound leveraging the dual form of Wasserstein Metric (Equation 2), and computing the fixed point of recurrence.
Theorem 4.3.
Given a HiP-MDP with learned ( K S ˆ R , K S ˆ T , K A ˆ R ,
K A ˆ T , K Ωˆ R , K Ωˆ T ) -Lipschitz transition dynamics ˆ T , reward function ˆ R , in model transfer , if γ ( K ST + K Sπ K AT ) < 1 , the generalized value function corresponding to the K Sπ -Lipschitz policy π ( a | s ) is Lipschitz continuous with respect to Ω with a constant bounded by:
[Equation]
Now if we further use the bounds derived in Theorem 3.1 to substitute K S Q π , we can get a bound of
K Ω V π that does not depend on K Q π .
Compared to Lemma 4.2, the bound derived in Theorem 4.3 does not depend on K Ω Q π by leveraging
Bellman equation (The details can be found in our proofs).
In general, Theorem 4.3 characterizes how the value estimation difference in model transfer is affected by the reward/transition functions’ robustness to the hidden parameter.
Theorem 4.3 and 3.1 together show that the value function of a
HiP-MDP for model transfer is Lipschitz continuous under certain conditions.
Then we can derive the regret of model transfer given hidden parameter estimation error ϵ .
Here we further assume the agent gets the optimal policy in the ”simulated environment” created by the learned transition and reward function.
We first introduce the following lemma:
Lemma 4.4.
Given a ( K SR , K ST , K Ω R , K Ω T ) -Lipschitz HiP-MDP, for the optimal policy π ∗ ,
[Equation]
Given the optimal policy, this lemma shows the relationship between value estimation difference and
Q-value estimation difference.
Combining the results in Corollary 3.2, we have:
Lemma 4.5.
Given a HiP-MDP with learned ( K S ˆ R , K S ˆ T , K Ωˆ R , K Ωˆ T ) -Lipschitz transition model ˆ T , reward function ˆ R , the optimal policy ˆ π ∗ ( a | s ) for ˆ T and ˆ R , starting from state distribution µ ( s ) ,if γK S ˆ T < 1 , the regret of model transfer given hidden parameter estimation error ϵ is bounded by:
[Equation]
Besides the reward and transition functions’ smoothness with respect to the hidden parameter θ , the derived bound also shows that the distance between the expected return of the learned policy ˆ π ∗ and the environment’s true optimal policy increases with K S ˆ R and K S ˆ T , given the hidden parameter estimation error ϵ .
In other words, the performance of model transfer algorithms decreases as the sensitivity of the dynamics changes in states increases, which aligns with the intuition that the model-based methods need to accurately predict state transitions and expected rewards given states.
Furthermore, for the theorems derived in this section, we show that (proofs included in the appendix):
Claim 4.6.
Given a linear and deterministic transition function, the bound derived in Theorem 4.1 is tight.
Given a linear and deterministic transition function and reward function, Lemma 4.5 is tight.
P ERFORMANCE BOUNDS OF P OLICY T RANSFER IN H I P-MDP S
We now focus on the regret bound induced by policy transfer algorithms in HiP-MDPs.
We assume the error bound of the hidden parameters is the same (i.e., d ( ω ( θ ) ,
ˆ ω ( θ )) ≤ ϵ ), and that the agent has learnt an accurate joint policy π ( a | s, ω ( θ )) during training.
Upon encountering a new task, the agent feeds the estimated hidden parameter into the learned policy instead of the model.
The agent will then directly use the joint policy to interact with the environment.
The algorithm framework can be found in Algorithm 1 and Figure 1.
Similar policy transfer paradigm has also been widely used (Yao et al., 2018; Rakelly et al., 2019).
We investigate how our estimation error for the hidden parameter will affect the performance of the joint policy on a new task.
Lemma 5.1.
Given a HiP-MDP with learned K Ω π -Lipschitz joint policy π ( a | s, ω ( θ )) , in policy transfer , the generalized value function is Lipschitz continuous with respect to Ω with a constant bounded by:
[Equation]
We obtain the above bound for the Lipschitz value function when assuming the Lipschitz continuity of the learned policy with respect to only the hidden parameter.
The lemma shows the relationship between K Ω V π and K Ω Q π in Policy transfer case.
Then, similar to model transfer, if we further assume the Lipschitz continuity of dynamics & policy and compute the fixed point of the recurrence of the
Bellman update, we can get the following bound:
Theorem 5.2.
Given a ( K AR , K AT , K SR , K ST ) -Lipschitz HiP-MDP, in policy transfer , the generalized value function corresponding to the pretrained ( K Sπ , K Ω π )-Lipschitz joint policy π ( a | s, ω ( θ )) , if γ ( K S T + K S π K A T ) < 1 , is Lipschitz continuous with respect to Ω with a constant bounded by
[Equation]
Now if we further use the bounds derived in Theorem 3.1 to substitute K AQ π , we can get a bound of K Ω V π that does not depend on K Q π .
Compared to Lemma 5.1, the bound derived in Theorem 5.2 does not depend on K Ω Q π by leveraging
Bellman equation.
Theorem 5.2 characterizes how the value estimation difference in policy transfer is affected by the policy’s robustness to the hidden parameter.
Theorem 5.2 and 3.1 together show the value function of a HiP-MDP for policy transfer is Lipschitz continuous under certain conditions.
Different from the bound derived in Theorem 4.3, for policy transfer methods this value estimation error is also affected by the policy’s robustness with respect to the hidden parameters, as well as the
Q-value’s robustness to the actions (only with respect to states in Theorem 4.3), with no dependence on the reward and transition functions’ smoothness with respect to the hidden parameters.
We can further derive the Lipschitz constant of the expected discounted reward (regret) starting from the state distribution µ .
Similar to what we did in model transfer case, if we also assume the learned joint policy over ω ( θ ) is optimal, by leveraging the results in Corollary 3.2, we have:
Lemma 5.3.
Given a ( K AR , K AT , K SR , K ST ) -Lipschitz HiP-MDP, a pretrained K Ω π -Lipschitz optimal joint policy π ( a | s, ω ( θ )) , starting from state distribution µ ( s ) , if γK ST ≤ 1 , the regret of policy transfer induced by hidden parameter estimation error ϵ is bounded by:
[Equation]
Comparing the derived bound with the one in Lemma 4.5, we find that the distance between the expected return of the learned joint policy π and the environment’s true optimal policy π ∗ increases also with K AT and K AR , besides K ST and K SR .
In other words, the performance of the policy transfer decreases as the sensitivity of the dynamics changes in both states and actions increases, while the model-based method is not quite sensitive to the dynamics change in actions.
This result is reasonable as policy-based method is influenced mostly by direct impact of actions.
Regarding the tightness of the derived bound, we get similar results as for model transfer:
Claim 5.4.
Given a linear and deterministic transition function, reward function and policy, the bounds derived in Lemma 5.3 are tight.
Summary Besides characterizing the robustness of model and policy transfer methods with respect to hidden parameter estimation error separately, Lemma 4.5 and 5.3 also imply that, policy transfer methods are expected to perform more robust when difference in effects of different actions is small, whereas model transfer methods are expected to perform better when neighboring states have relatively similar dynamics.
Given these bounds, one direct implication to HiP-MDP algorithms in practice is that we can infer the performance trend of policy transfer and model transfer algorithms by either qualitatively or quantitatively estimating the dynamics and reward functions’ sensitivity to states and actions.
This can further help us determine which method is probably more advantageous for a specific HiP-MDP problem.
Note that K θ ˆ T and K Ω π in general can be dependent on the model class (architecture) of the model (e.g. neural nets) used to parameterized the policy or model.
E MPIRICAL E VALUATION
We now corroborate our theoretical results in relatively large scenarios where neural networks are needed to parameterize models and polices.
We evaluate model and policy transfer methods on two continuous scenarios, ball-goal and ball-wind , where we can quantitatively estimate the influence of changing environment hyperparameters on different Lipschitz constants.
Then by investigating whether the empirical performance changing is consistent with what we expect from theoretical results, we can corroborate whether our derived theorems match what happened in practice.
For both scenarios, the goal of the agent is to control a ball to quickly reach a target position.
In ball-goal , the angle of the goal direction is the hidden parameter.
That is, the goal direction changes across tasks, and the agent will obtain maximum cumulative reward if it figures out the right angle for the current task and keeps moving in that direction until reaching the goal.
In ball-wind , the goal position is fixed across different tasks but there is wind across the plane with different directions.
The angle of the wind direction is the hidden parameter in this scenario.
The reward function consists of a dense reward proportional to how closer the ball moves towards the goal compared to last step, a control cost, as well as a larger final reward for reaching the goal.
Thus for ball-goal , only the reward function is changing with respect to different hidden parameters, while in ball-wind , both reward and transition functions are changing.
More details about the environments can be found in Appendix B.
For each HiP-MDP, we manually calculate the value of different Lipschitz constants approximately given the actual reward and transition functions for both scenarios.
We further calculate the bounds derived in Lemma 4.5 and Lemma 5.3 using the results.
The Lipschitz constants are given in Table 1.
We mainly investigate the effect of the step size v , goal distance g , and state accelerator m ( ball-wind ).
The results imply how different Lipschitz constants are changing as we change those environmental hyper-parameters if our derived bounds are relatively tight and close to what happened in practice.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
A v e r a g e R e t u r n
Ball wind ( v ) 8 9 10 11 12 13 14 15 16 g 80 7565 60 55
K J
Model transfer 8 9 10 11 12 13 14 15 16 g 150130 120 110
K J
Policy transfer 0.8 1.0 1.2 1.4 1.6 1.8 v51
K J 0.8 1.0 1.2 1.4 1.6 1.8 v 11090 80
K J 6 7 8 9 10 11 12 13 14 g 9.
K J 6 7 8 9 10 11 12 13 14 g 119 8
K J 0.4 0.5 0.6 0.7 0.8 0.9 v 11.0 10.5 10.
9.5 9.0 8.5 8.0 7.5 K J 0.4 0.5 0.6 0.7 0.8 0.9 v 8.6 8.
8.2 8.0 7.8 K J
6. E MPIRICAL RESULTS CORRESPONDING TO L EMMA 4.5 AND L EMMA 5.
As both Lemma 4.5 and Lemma 5.3 can be written in the form | J ˆ π − J π ∗ | < K J · ϵ , we investigate how K J is changing as we change different environmental hyperparameters both theoretically : manually computing the Lipschitz constants given the environment’s dynamics (appendix B) and the two lemmas (results shown in Table 1), and empirically : investigating how the algorithm’s empirical performance is changing as we alter the environment’s hyperparameters ( K J is expected to inversely related to the expected return), and see whether the theoretical and empirical results are consistent.
For empirical training, we use state-of-the-art meta-RL algorithms PEARL (Rakelly et al., 2019) as the policy transfer method to learn the joint policy, as well as CaDM (Lee et al., 2020) to learn the dynamics model.
The context encoder (probabilistic encoder) is updated together with loss from
Q-value prediction like in PEARL, as well as loss from dynamics model prediction like in CaDM.
Given a new task, we let the agent collect transitions by interacting with the environment for only one episode with randomly initialized hidden-parameter distribution.
Then the agent will use the learned context encoder to infer a distribution over the hidden parameter of the current task.
This distribution over the hidden parameter is fixed afterwards for the current task, i.e. ϵ is fixed and the same for both methods.
Then, for policy transfer, we sample from the hidden parameter distribution and directly feed that inferred latent parameter into the learned joint policy.
We evaluate the average return of this policy on the current task.
For model transfer, we feed that inferred latent parameter into the learned transition model and reward function.
We use the models to generate samples and run SAC (Haarnoja et al., 2018) to learn the task-specific policy for this “simulated environment”.
The empirical results are shown in Figure 2.
We plot − K J v.s. v/g using the results in Table 1.
Recall that K J is expected to inversely related to the average return.
i.e., increase in − K J implies a decrease in the regret of the algorithm, thus the average return is expected to increase if our theoretical results match what happened in practice.
Almost all of the results show that the change of the expected return is inversely related to the change of the approximated Lipschitz constant K J .
The environment parameters directly affect K J and the trend of the performance change in a way that is highly consistent with our theoretical analysis.
In general, we find that a slower increase in
Lipschitz constant with respect to the hidden parameter implies a smaller performance decay .
The empirical results implies that for real-world HiP-MDP tasks, if we can quantitatively estimate the Lipschitz constant K J for model and policy transfer methods given the environment parameters, we can estimate which one is expected to have better final performance.
For instance, in ball-wind, if g is quite large, given the computed K J , we will expect model transfer to have better performance as it is less affected by the value of g compared to policy transfer method.
Note that in our scenarios, the performance drop is mainly induced by the estimation error of the hidden parameters, and the learned dynamics and policies are near optimal.
We show the empirical evidence in appendix C.
6. E MPIRICAL RESULTS CORRESPONDING TO T HEOREM 4.
[Figure]
We also show how the bound of the multi-step prediction error derived in Theorem 4.1 is related to the performance of the other planning method used in model transfer — Model Predictive Control (MPC) (Garcia et al., 1989).
MPC is widely used in recent deep model-based
RL works (Chua et al., 2018; Nagabandi et al., 2019).
Different from the previous approaches,
MPC does not need to explicitly estimate the value function and its performance is directly affected by the multi-step prediction error.
With the results in Table 1, we can get the multi-step prediction upper bound for ball-wind using Theorem 4.1: ξ ( n ) ≤ v (cid:80) n − 1 i =
m i · ϵ .
The bound implies that if the theorem matches what happens in practice, the n -step prediction error will increase as we increase v and m , and thus the performance is expected to drop as we increase v and m .(The
performance of MPC and the multi-step prediction error are inversely correlated.)
Similar to the previous subsection, we investigate whether the empirical performance of MPC matches our predictions from theoretical results.
During empirical evaluation, we first let the agent infer the hidden parameter and feed that into the learned transition function and reward function.
Then at each time step, we let the agent randomly sample a large number of actions, and use the learned model to predict N steps into the future for each of them and choose the action with the highest predicted cumulative reward.
We investigate how v and m affect the algorithm’s final performance empirically.
The results are shown in Figure 3, the average return of the algorithm on test tasks decreases as we increase the value of v and m , which is consistent with our theoretical result.
We also find that the variance of the performance (the width of error bar) is increasing as we use longer planning horizon for MPC.
C ONCLUSION
Assuming the learned model/policy is optimal, we discuss how the hidden parameter estimation error affects the robustness of model and policy transfer algorithms from the theoretical perspective, respectively.
We show that under which conditions the value functions of HiP-MDPs are Lipschitz continuous.
We further derive regret bounds for model and policy transfer, which are proved to be tight in linear and deterministic cases.
Our empirical results are consistent with the theoretical results and indicate that a faster increase in Lipschitz constant with respect to the hidden parameter implies a larger performance decay.
We note that in real-world HiP-MDP problems, especially when applying deep RL algorithms, the suboptimality of the learned model/policy besides the estimation error of the hidden parameter can still play an important role in the potential performance drop on a new task.
And in many cases, policy transfer performs better than model transfer simply because the downstream model-free deep RL algorithms are consistently better in regular MDPs.
A M ODEL TRANSFER AND P OLICY TRANSFER
Input: Test tasks { τ i } i =1: K with hidden parameters θ i ∼ P Ω for each task τ
i do
Roll out policy π ( a | s,
˜ ω
( θ )) with randomly generated Gaussian ˜ ω
( θ ) to collect data c = { s 1 , a 1 , r 1 , · · · , s n , a n , r n }
Infer the distribution ˆ ω ( θ ) over the hidden parameter of task τ
i with the learned encoder q ( ω ( θ ) | c )
Roll out policy π ( a | s, ˆ ω ( θ )) to interact with environment ( evaluation ) end for
Input: Test tasks { τ i } i =1: K with hidden parameters θ i ∼ P Ω for each task τ
i do
Roll out policy π ( a | s,
˜ ω
( θ )) with randomly generated Gaussian ˜ ω
( θ ) to collect data c = { s 1 , a 1 , r 1 , · · · , s n , a n , r n }
Infer the distribution ˆ ω ( θ ) over the hidden parameter of task τ
i with the learned encoder q ( ω ( θ ) | c )
Planning with learned transition function T g ( s ′ | s, a, ˆ ω ( θ )) and reward function R g ( s, a, , ˆ ω ( θ )) ( evaluation ) end for
Empirically, in typical deep meta-RL settings (Finn et al., 2017; Rakelly et al., 2019), the true value of hidden parameters are not known to the agent both during training phase and evaluation phase.
We follow this setting in out experiments and train a context encoder like the previous approaches to infer the hidden parameter in latent representation space .
Then the true distribution over hidden parameter is also assumed to be implicitly mapped into the latent representation space.
Our empirical experiments show that our theories can be extended to hidden parameter estimation in the latent space, and the results show consistent patterns with the theoretical results.
B E XPERIMENT SETTINGS
Ball-goal : The state space consists of the ball’s x, y coordinates.
The action space consists of the ball’s x, y velocity.
v controls the step size in the direction of x axis in the transition function.
The reward is proportional to the how far the agent has moved towards the goal’s position ( g cos ϕ g , g sin ϕ g ) .
We fix the value of v, g and use ϕ g as the hidden parameter to create HiP-MDPs.
Details are shown below:
• State space: { s x , s y } • Action space: { a x , a y } • Transition function:
[Equation]
• Reward function:
[Equation]
Ball-wind :The state space consists of the ball’s x, y coordinates.
The action (one dimension) describes the direction of the agent’s next move.
In the transition function, m is describes the value of state accelerator, v controls the step size of the wind, θ describes the direction of the wind.
The reward is proportional to the how far the agent has moved towards the goal’s position ( g cos ϕ g , g sin ϕ g ) , plus a goal reward when succeeds and a control penalty ( − 0 . 5 ).
We fix the value of v, g, m, ϕ g and use θ as the hidden parameter to create HiP-MDPs.
Details are shown below:
• State space: { s x , s y } • Action space: { a } • Transition function:
[Equation]
• Reward function:
[Equation]
When computing the Lipschitz constants for Ball-wind [Figure] we only consider the difference of distance towards goal part in the reward function for ease of calculation [Figure]
C A DDITIONAL EXPERIMENTS WHEN THE HIDDEN PARAMETER IS BETTER INFERRED
[Figure]
We further show that in these two scenarios, the performance drop as we change environment parameters is mainly induced by the estimation error of the hidden parameters, and the learned transition/reward functions and policies are near optimal.
As shown in Figure 5, when we let the agent interact with the environments for more trajectories and keep using the collected data to infer the hidden parameter, changing of the environment’s
Lipschitz constant does not affect the performance as much as before when the estimation of the hidden parameter is far less accurate.
D C OMPARISON OF T HEOREM 3.1 WITH P REVIOUS RESULTS
Assumption 1 . The policy is optimal. Assumption 2 .
S and A share the same Lipschitz constant.
[Table]
Theorem 3.1 and Corollary 3.2 also directly apply to a ( K SR , K ST , K AR , K AT ) -Lipschitz regular MDP with a K Sπ -Lipschitz policy, as the Lipschitz continuity assumptions with respect to hidden parameters are not used in both the final results and proofs.
In this sense, we show a comparison of Theorem 3.
with previous results in regular MDPs in Table 2.
Note that in Theorem 3.1, we have:
[Equation]
Specifically, the upper bound for K SQ ∗ in Corollary 3.2 recovers the results in (Asadi et al., 2018;
Gelada et al., 2019).
If we assume the Lipschitz constants for state and action are the same, we get
Corollary E.1, which recovers the results in (Rachelson & Lagoudakis, 2010; Pirotta et al., 2015).
For regular MDPs, our results are more general compared to prior works as (1) we do not assume that the Lipschitz constants for state and action are the same because this is usually not the case in practice; (2) our theory applies to all Lipschitz policies rather than only the optimal policy.
E P ROOFS AND MORE THEORY RESULTS
Using the generalized functions defined in Section 2, we can easily extend Bellman update rule to HiP-MDPs:
[Equation]
where Q n +1 converges to Q ∗ as n → ∞ .
Theorem 3.1 .For a ( K SR , K ST , K AR , K AT , K Ω R , K Ω T ) -Lipschitz HiP-MDP with a ( K Sπ , K Ω π ) -Lipschitz
policy π , if γ ( K ST + K Sπ K AT ) < 1 , the value function is Lipschitz continuous with respect to S & A with constants bounded by:
[Equation]
Proof. The proof is mainly based on the Bellman update rule (9), the dual form of Wasserstein Metric (Equation 2)
[Equation]
and fixed point iteration.
Recall that:
[Equation]
Now let:
[Equation]
The last inequality holds according to Equation
[Equation]
Similarly, we can get:
[Equation]
We need to further derive the upper bound of K V :
[Equation]
Plugging Eqn 12 into Eqn 10& Eqn 11, we get:
[Equation]
By computing the fixed point of the recurrence [Equation] we get:
[Equation]
Corollary E.1.
If we assume the Lipschitz constants for state and action are the same ( K AR =
K SR =
K S,AR , K AT =
K ST = K S,AR = K S,AT ) : | R g ( s 1 , a 1 , ω ) − R g ( s 2 , a 2 , ω ) | ≤ K S,AR d s,a (( s 1 , a 1 ) , ( s 2 , a 2 )) [Equation] [Equation]
, a [Equation] , ω [Equation]
, T g
[Equation]
then:
[Equation]
Proof.
Let K A R =
K S R = K S,A R , K A T = K S T = K S,A R = K S,A T in the bounds of Theorem 3.1.
Corollary 3.2 .
If the policy π is optimal and γK ST < 1 ,then:
[Equation]
Proof.
When the policy is optimal, we have (Asadi et al., 2018; Gelada et al., 2019):
[Equation]
Thus,
[Equation]
Plugging Eqn 17 into Eqn 10 and Eqn 11, we get:
[Equation]
First compute the fixed point of the recurrence of Eqn 19, we get:
[Equation]
Now plugging Eqn 20 back into Eqn 18:
[Equation]
Definition E.2.
Given a metric state space ( S, d S ) , an action space A , and a metric hidden-parameter space (Θ , d Θ ) , we define F g as a collection of functions: F g = { f : S × Θ (cid:55)→ S } distributed according to g ( f | a ) where a ∈ A .
We say that ( F Sg , F Θ g ) is a Lipschitz model class in HiP-MDPs if
[Equation]
(22) and
[Equation]
are finite.
The transition function associated with a Lipschitz model class then can be defined by:
[Equation]
And the generalized transition function associated with a Lipschitz model class would be:
[Equation]
We introduce the following two lemmas regarding Lipschitz model class (Asadi et al., 2018)(We also include a definition in appendix):
Lemma E.3.
(Asadi et al., 2018)
A generalized transition function T g induced by a Lipschitz model class F g and fixed ( θ, a ) is Lipschitz with a constant:
[Equation]
Lemma E.4.
A generalized transition function T g induced by a Lipschitz model class F g and fixed ( s, a ) is Lipschitz with a constant:
[Equation]
Intuitively, the above Lemmas give a bound on the differences in distributions an agent transitions to, as a function of how different the distributions of states it transitions from are, or how different the distributions of the hidden parameters are.
It is theoretically pleasing that these bounds are given as the Lipschitz constants for the corresponding differences between point distributions of either states or hidden parameters [Equation]
Proof for Lemma E
[Equation]
Proof.
[Equation]
Using Lemma E.3, we can derive the bound of the compounding error of dynamics prediction given the estimation error ϵ in hidden parameter at test time (Theorem 4.1).
Theorem 4.1 .
Assuming the estimation for hidden parameter θ is bounded by ϵ , that is,
W ( ω ( θ ) , ˆ ω ( θ )) ≤ ϵ , an learned accurate ˆ T g induced by a Lipschitz model class F g with the Lipschitz constant K Ωˆ T and K S ˆ T , a fixed sequence of actions a 0 , · · · , a n − 1 , and a start state distribution µ .
Then ∀ n ≥ 1 :
[Equation]
(29) where ˆ T ng ( ·| µ, ω )) :
=
ˆ T g ( ·| ˆ T g ( ·| ...
ˆ T g ( ·| µ, a 0 , ω ) ..., a n − 2 , ω ) , a n − 1 , ω ) , and ˆ T ng ( ·| µ, ˆ ω )) is defined similarly.
Proof.
We first derive the bound for one-step prediction error given Lemma E.3.
[Equation]
Then
[Equation]
we can prove the bound for ξ ( n ) : ξ ( n ) : = W ( ˆ T ng ( ·| µ, ω ) , ˆ T ng ( ·| µ, ˆ ω ))
[Equation]
We can get the same results if we replace ˆ T ng ( ·| ˆ T n − 1 g ( ·| µ, ˆ ω ) , ω ) with ˆ T ng ( ·| ˆ T n − 1 g ( ·| µ, ω ) , ˆ ω ) in the triangle inequality.
Lemma 4.2 .Given a HiP-MDP with learned ( K Ωˆ R , K Ωˆ T ) -Lipschitz transition model ˆ T [Equation] ˆ R , in Model
Transfer , the generalized value function is Lipschitz continuous with respect to Ω with a constant bounded by:
[Equation]
Proof.
Recall that:
[Equation]
Then we have:
[Equation]
Recall that we quantify the distance between ω 1 ( θ ) and ω 2 ( θ ) using Wasserstein Metric W ( ω 1 , ω 2 ) , then
[Equation]
Theorem 4.3 .Given a HiP-MDP with learned ( K S ˆ R , K S ˆ T , K A ˆ R , K A ˆ T , K Ωˆ R , K Ωˆ T ) -Lipschitz
transition model ˆ T , ˆ R , in Model Transfer , if γ ( K S T + K S π K A T ) < 1 , the generalized value function corresponding to the K Sπ -Lipschitz policy π ( a | s ) is Lipschitz continuous with respect to Ω with a constant bounded by:
[Equation]
Proof.
By further assuming the Lipschitz continuity of dynamics & policy in Lemma 4.2, we can get the result bound leveraging the dual form of Wasserstein Metric (Equation 2), triangle inequality, and computing the fixed point of recurrence.
Recall that:
[Equation]
a ∼
[Equation]
It is well known that as n → ∞ , Q n +1 converges to Q ∗ , now let:
[Equation]
Recall that we quantify the distance between ω 1 ( θ ) and ω 2 ( θ ) using Wasserstein Metric W ( ω 1 , ω 2 ) ,
[Equation]
Recall the bound for ( K S ˆ R , K S ˆ T ) -lipschitz
[-ds ′ 1 | W ( ω 1 , ω 2 ) + γ sup s ∈ S sup a ∈ A sup ω 1 ,ω| (cid:82) s ′ 1 ˆ T g ( s ′ 1 | s, a, ω 1 ) Q πn ( s ′ 1 , a, ω 2 ) ds ′ 1 − (cid:82) s ′ 
 ˆ T g ( s ′ 2 | s,-]MDP:
K SQ
[Equation]
sup
[Equation]
Equivalently:
[Equation]
[Equation]
By computing the limit of both sides:
[Equation]
From Lemma 4.2 we have:
[Equation]
Thus,
[Equation]
Lemma 4.4 .
Given a ( K SR , K ST , K Ω R , K Ω T ) -Lipschitz HiP-MDP, for the optimal policy π ∗ , |
[Equation]
Proof.
[Equation]
Lemma 4.5 .Given a HiP-MDP with learned ( K S ˆ R , K S ˆ T , K Ωˆ R , K Ωˆ T ) -Lipschitz transition model
ˆ T , ˆ R , and the optimal policy ˆ π ∗ ( a | s ) for ˆ T and ˆ R , starting from state distribution µ , the expected discounted reward difference induced by hidden parameter estimation error ϵ is bounded by:
[Equation]
if γK S ˆ T < 1 .
Proof.
Recall that:
[Equation]
We have:
[Equation]
From Corollary [Equation] [Equation] we have:
[Equation]
Thus,
[Equation]
Claim 4.6 .
Given a linear and deterministic transition function and reward function, the bound derived in Theorem 4.1 & Lemma 4.5 are tight.
Proof.
Assume a linear transition function T for a new task with hidden parameter θ defined as:
[Equation]
And a reward function defined as:
[Equation]
Assume there’s an estimation error ϵ for θ , then the transition function and reward function becomes:
[Equation]
First observe that:
[Equation]
And that for n = 2 :
[Equation]
More generally, for n step compounding error of dynamics prediction T n and ˆ T n , given a fixed sequence of actions a 0 , a 1 , · · · , a n :
[Equation]
Thus, the bound in Theorem 4.1 is tight.
Now consider the state s = 0 and the action space only consists of one action a = 0 (thus the policy is optimal), we can calculate the value of s predicted with ˆ T and ˆ R :
[Equation]
Thus:
[Equation]
The result matches the bound derived in Lemma 4.5.
(In deterministic cases, we assume we directly estimate the value of θ , so K Ω T & K Ω R becomes K Θ T & K Θ R in this case and the Wasserstein distance becomes the L1 distance | ˆ θ − θ | .)
Lemma
[Equation]
Given a HiP-MDP with learned K Ω π -Lipschitz joint policy π ( a | s, ω ( θ )) , in Policy
Transfer , the generalized value function with respect to Ω with a constant bounded by:
[Equation]
Proof.
Given a new test task
[Equation]
recall that:
[Equation]
where we use ϕ to denote the true distribution ϕ ( θ ) over hidden parameter for the current task (which should be a Dirac δ -function in practice) of the environment
[Equation]
Then we have:
[Equation]
Theorem 5.2 .
Given a ( K A R , K A T , K S R , K S T ) -Lipschitz HiP-MDP, in Policy Transfer , the generalized value function corresponding to the pretrained ( K Sπ , K Ω π )-Lipschitz joint policy π ( a | s, ω ( θ )) , if γ ( K ST + K Sπ K AT ) < 1 , is Lipschitz continuous with respect to Ω with a constant bounded by
[Equation]
[Equation]
Proof.
Similar to model transfer, if we further assume the Lipschitz continuity of dynamics & policy in Lemma 5.1, we can get the result bound leveraging the dual form of Wasserstein Metric (Equation 2), triangle inequality, and computing the fixed point of recurrence.
Recall that:
[Equation]
[Table]
[Equation]
Now let:
[Equation]
Equivalently:
[Equation]
Computing the limit of both sides:
[Equation]
Plugging the results back in to Equation
[Equation]
we have:
[Equation]
Lemma 5.3 .
Given a ( K AR , K AT , K SR , K ST ) -Lipschitz HiP-MDP, a pretrained K Ω π -Lipschitz optimal joint policy π ( a | s, ω ( θ )) , starting from state distribution µ , the expected discounted reward difference induced by hidden parameter estimation error ϵ is bounded by:
[Equation]
if γK ST ≤
[Equation]
Proof.
When the policy is optimal
[Equation]
we have:
[Equation]
Equivalently:
[Equation]
Computing the limit of both sides:
[Equation]
If the policy is optimal
[Equation]
we have:
[Equation]
Then for the Lipschitz constant of the expected return function
[Equation]
we have:
[Equation]
Using the results from Corollary 3.2, we get:
[Equation]
Claim 5.4 .
Given a linear and deterministic transition function, reward function and policy, the bounds derived in Lemma 5.3 are tight.
Proof.
Assume a linear transition function T for a new task defined as:
[Equation]
49) a reward function defined as:
[Equation]
And the optimal policy corresponding to hidden parameter θ defined as:
[Equation]
Assume there’s an estimation error ϵ for θ , then the policy becomes:
[Equation]
Now consider the state s = 0 , we can calculate the value of s predicted with T , R and ˆ π : ˆ V
[Equation]
Thus:
[Equation]
The result matches the bound derived in Lemma 5.3.
(In deterministic cases, we assume we directly estimate the value of θ , so K Ω π becomes K Θ π in this case and the Wasserstein distance becomes the
L1 distance | ˆ θ − θ | .)
F F URTHER CLAIM ABOUT THE ASSUMPTION γK YX <
Similar assumptions have been made in most of previous papers discussing Lipschitz continuity in
RL (Rachelson & Lagoudakis, 2010; Pirotta et al., 2015; Asadi et al., 2018; Gelada et al., 2019).
Here, we make the following claim:
Claim F.1.
When a HiP-MDP has a fixed horizon N , the assumption that γ ( K ST + K Sπ K AT ) <
1 and the assumption (when the policy is optimal) γK ST < 1 are both unnecessary.
Proof.
[Equation]
Firstly, if the agent is at the last step N of one episode:
[Equation]
If the agent is at step N − 1 :
[Equation]
If the agent is at step N − 2 , similarly, we have:
[Equation]
By induction, if the agent is at step N − n
[Equation]
we have:
[Equation]
Thus, the assumption that γ ( K ST + K Sπ K AT ) < 1 is unnecessary.
Similarly, when the policy is optimal, at step N − 1 :
[Equation]
At step N − n , we have:
[Equation]
Thus, the assumption that γK ST < 1 is unnecessary.
Intuitively, for tasks with infinite horizons, the assumption requires that the future states generated by close states are not too divergent.
The threshold of “divergent” depends on how farsighted the agent is.