CUP: Critic-Guided Policy Reuse
Anonymous Author(s)
AffiliationAddress email
Abstract
The ability to reuse previous policies is an important aspect of human intelligence.
To achieve efficient policy reuse, a Deep Reinforcement Learning (DRL)agent needs to decide when to reuse and which source policies to reuse.
Previousmethods solve this problem by introducing extra components to the underlyingalgorithm, such as hierarchical high-level policies over source policies, or estimations of source policies’ value functions on the target task.
However, training thesecomponents induces either optimization non-stationarity or heavy sampling cost,significantly impairing the effectiveness of transfer.
To tackle this problem, wepropose a novel policy reuse algorithm called Critic-gUided Policy reuse (CUP),which avoids training any extra components and efficiently reuses source policies.
CUP utilizes the critic, a common component in actor-critic methods, to evaluateand choose source policies.
At each state, CUP chooses the source policy thathas the largest one-step improvement over the current target policy, and forms aguidance policy.
The guidance policy is theoretically guaranteed to be a monotonicimprovement over the current target policy.
Then the target policy is regularized toimitate the guidance policy to perform efficient policy search.
Empirical resultsdemonstrate that CUP achieves efficient transfer and significantly outperformsbaseline algorithms.
Introduction
Human intelligence can solve new tasks quickly by reusing previous policies (Guberman & Greenfield,1991).
Despite remarkable success, current Deep Reinforcement Learning (DRL) agents lack thisknowledge transfer ability (Silver et al., 2017; Vinyals et al., 2019; Ceron & Castro, 2021), leading toenormous computation and sampling cost.
As a consequence, a large number of works have beenstudying the problem of policy reuse in DRL, i e., how to efficiently reuse source policies to speed uptarget policy learning (Fernández & Veloso, 2006; Barreto et al., 2018;
Li et al., 2019; Yang et al.,2020b).
A fundamental challenge towards policy reuse is: how does an agent with access to multiple sourcepolicies decide when and where to use them (Fernández & Veloso, 2006; Kurenkov et al., 2020;
Cheng et al., 2020)?
Previous methods solve this problem by introducing additional components tothe underlying DRL algorithm, such as hierarchical high-level policies over source policies (Li et al.
Yang et al., 2020b), or estimations of source policies’ value functions on the target task(Barreto et al., 2017, 2018; Cheng et al., 2020).
However, training these components significantlyimpairs the effectiveness of transfer, as hierarchical structures induce optimization non-stationarity(Pateria et al., 2021), and estimating the value functions for every source policy is computationallyexpensive and with high sampling cost.
Thus, the objective of this study is to address the question
Can we achieve efficient transfer without training additional components?
Notice that actor-critic methods (Lillicrap et al., 2016; Fujimoto et al., 2018;
Haarnoja et al., 2018)learn a critic that approximates the actor’s Q function and serves as a natural way to evaluate policies.
Based on this observation, we propose a novel policy reuse algorithm that utilizes the critic tochoose source policies.
The proposed algorithm, called Critic-gUided Policy reuse (CUP) , avoidstraining any additional components and achieves efficient transfer.
At each state, CUP chooses thesource policy that has the largest one-step improvement over the current target policy, thus forming aguidance policy.
Then CUP guides learning by regularizing the target policy to imitate the guidancepolicy.
This approach has the following advantages.
First, the one-step improvement can be estimatedsimply by querying the critic, and no additional components are needed to be trained.
Secondly, theguidance policy is theoretically guaranteed to be a monotonic improvement over the current targetpolicy, which ensures that CUP can reuse the source policies to improve the current target policy.
Finally, CUP is conceptually simple and easy to implement, introducing very few hyper-parametersto the underlying algorithm.
We evaluate CUP on Meta-World (Yu et al., 2020), a popular reinforcement learning benchmarkcomposed of multiple robot arm manipulation tasks.
Empirical results demonstrate that CUP achievesefficient transfer and significantly outperforms baseline algorithms.
Preliminaries
Reinforcement learning (RL) deals with Markov Decision Processes (MDPs).
A MDP can bemodelled by a tuple ( S , A , r, p, γ ) , with state space S , action space A , reward function r ( s, a ) ,transition function p ( s ′ | s, a ) , and discount factor γ (Sutton & Barto, 2018).
In this study, we focuson MDPs with continuous action spaces.
RL’s objective is to find a policy π ( a | s ) that maximizes thecumulative discounted return R ( π ) =
E π [ (cid:80) ∞ t =
γ t r ( s t , a t )] .
While CUP is generally applicable to a wide range of actor-critic algorithms, in this work we use
SAC (Haarnoja et al., 2018) as the underlying algorithm.
The soft Q function and soft V function(Haarnoja et al., 2017) of a policy π are defined as:
[Equation]
where α > 0 is the entropy weight.
SAC’s loss functions are defined as:
[Equation]
where D is the replay buffer, H is a hyper-parameter representing the target entropy, θ andϕ are network parameters, θ is target network’s parameters, and V θ ( s ) =
E a ∼ π ( a | s )
[ Q θ ( s, a ) −α log π ( a | s )] is the target soft value function.
We define the soft expected advantage of action probability distribution π i ( ·| s ) over policy π j at states as:
[Equation]
EA π j ( s, π i ) measures the one-step performance improvement brought by following π i instead ofπ j at state s , and following π j afterwards.
The field of policy reuse focuses on solving a target
MDP M efficiently by transferring knowledgefrom a set of source policies { π 1 , π 2 , ..., π n } .
We denote the target policy learned on M at iterationt as π ttar , and its corresponding soft Q function as Q π ttar .
In this work, we assume that the sourcepolicies and the target policy share the same state and action spaces.
Critic-Guided Policy Reuse
This section presents CUP, an efficient policy reuse algorithm that does not require training anyadditional components.
CUP is built upon actor-critic methods.
In each iteration, CUP uses the criticto form a guidance policy from the source policies and the current target policy.
Then CUP guidespolicy search by regularizing the target policy to imitate the guidance policy.
Section 3.1 presentshow to form a guidance policy by aggregating source policies through the critic, and proves that theguidance policy is guaranteed to be a monotonic improvement over the current target policy.
We alsoprove that the target policy is theoretically guaranteed to improve by imitating the guidance policy.
Section 3.2 presents the overall framework of CUP.
3.1 Critic-Guided Source Policy Aggregation
CUP utilizes action probabilities proposed by source policies to improve the current target policy, andforms a guidance policy.
At iteration t of target policy learning, for each state s , the agent has accessto a set of candidate action probability distributions proposed by the n source policies and the currenttarget policy: Π st = { π 1 ( ·| s ) , π 2 ( ·| s ) , ..., π n ( ·| s ) , π ttar ( ·| s ) } .
The guidance policy π tg can be formedby combining the action probability distributions that have the largest soft expected advantage overπ ttar at each state s :
[Equation]
The second equation holds as adding V π ttar ( s ) to all soft expected advantages does not affect theresult of the arg max operator.
Eq.
5 implies that at each state, we can choose which source policyto follow simply by querying its expected soft Q value under π ttar .
Noticing that with functionapproximation, the exact soft Q value cannot be acquired.
The following theorem enables us to formthe guidance policy with an approximated soft Q function, and guarantees that the guidance policy isa monotonic improvement over the current target policy.
Theorem 1 Let (cid:101)
Q π ttar be an approximation of Q π ttar such that
[Equation]
Define
[Equation]
Then,
[Equation] [Equation]
Theorem 1 provides a way to choose source policies using an approximation of the current targetpolicy’s soft Q value.
As SAC learns such an approximation, the guidance policy can be formedwithout training any additional components.
The next question is, how to incorporate the guidance policy (cid:102)
π tg into target policy learning?
Thefollowing theorem demonstrates that policy improvement can be guaranteed if the target policy isoptimized to stay close to the guidance policy.
Theorem
If
[Equation] [Equation]
then
[Equation] [Equation]
where (cid:101) R max = max s,a | r ( s, a ) | is the largest possible absolute value of the reward, H t +1 max =max s H ( π t +1 tar ( ·| s )) is the largest entropy of π t +1 tar , and (cid:101) H max = max s (cid:12)(cid:12) H ( π ttar ( ·| s ))
− H ( π t +1 tar ( ·| s ))
(cid:12)(cid:12)is the largest possible absolute difference of the policy entropy.
According to Theorem 2, the target policy can be improved by minimizing the KL divergencebetween the target policy and the guidance policy.
Thus we can use the KL divergence as an auxiliaryloss to guide target policy learning.
Proofs of this section are deferred to Appendix A.1 and Appendix
A.2.
Theorem 1 and Theorem 2 can be extended to common “hard” value functions (deferred to
Appendix A.3), so CUP is also applicable to actor-critic algorithms that uses “hard” Bellman updates,such as A3C (Mnih et al., 2016).
3.2 CUP Framework
[Figure]
In this subsection we propose the overall framework of CUP.
As shown in Fig.
1, at each iteration t ,
CUP first forms a guidance policy (cid:102)
π tg according to Eq.
7, then provides additional guidance to policysearch by regularizing the target policy π t +1 tar to imitate (cid:102)
π tg (Wu et al., 2019; Fujimoto & Gu, 2021).
Specifically, CUP minimizes the following loss to optimize π t +1 tar :
[Equation]
where L actor is the original actor loss defined in Eq.
(3), and β s > 0 is a hyper-parameter controllingthe weight of regularization.
In practice, we find that using a fixed weight for regularization has twoproblems.
First, it is difficult to balance the scale between L actor and the regularization term, because
L actor grows as the Q value gets larger.
Secondly, a fixed weight cannot reflect the agent’s confidenceon (cid:102)
π tg .
For example, when no source policies have positive soft expected advantages, (cid:102) π tg = π ttar .
Then the agent should not imitate (cid:102)
π tg anymore, as (cid:102)
π tg cannot provide any guidance to further improveperformance.
Noticing that the soft expected advantage serves as a natural confidence measure, weweight the KL divergence with corresponding soft expected advantage at that state:
[Equation]
(12) where (cid:103) EA π ttar ( s, (cid:102)
π tg )
=
E
a ∼ (cid:102)
π tg ( ·| s ) (cid:104) (cid:101) Q π ttar ( s, a ) − α log π tg ( a | s ) − (cid:101)
V π ttar ( s ) (cid:105) is the approximated soft expected advantage, β 1 , β 2 > 0 are two hyper-parameters, and (cid:101)
V π ttar ( s ) =
E a ∼ π t tar ( ·| s ) (cid:104) (cid:101) Q π t tar ( s, a ) − α log π t tar ( a | s ) (cid:105) is the approximated soft value function.
This adaptiveregularization weight automatically balances between the two losses, and ignores the regularizationterm at states where (cid:102)
π tg cannot improve over π ttar anymore.
We further upper clip the expectedadvantage with the absolute value of β 2 (cid:101)
V π ttar to avoid the agent being overly confident about (cid:102) π tg dueto function approximation error ϵ .
Algorithm
Require: Source policies { π 1 , π 2 , ..., π n } , hyper-parameters λ θ 1 , λ θ 2 , λ π , λ α , τ, H , β 1 , β
Initialize replay buffer D
Initialize actor π ϕ , entropy weight α , critic Q θ 1 , Q θ 2 , target networks Q θ 1 ← Q θ 1 , Q θ 2 ← Q θwhile not done do for each environment step do a t ∼ π θ s t +1 ∼ p
( s t +1 | s t , a t )
D ← D ∪ { s t , a t , r ( s t , a t ) , s t +1 } end for for each gradient step do
Sample minibatch b from D
Query source policies’ action probabilities { π 1 ( ·| s ) , π 2 ( ·| s ) , ..., π n ( ·| s ) } for states in b
Compute expected advantages according to Eq.
(4), form (cid:102)
π tg according to Eq.
(7) θ i ← θ i − λ Q ˆ ∇ θ i L critic ( Q θ i ) for i ∈ { 1 , 2 }
ϕ ← ϕ − λ π ˆ ∇
ϕ
L CUP ( π ϕ )
α ← α − λ α ˆ ∇ α L entropy ( α ) θ
i ←
τθ i + (1 − τ ) θ i for i ∈ { 1 , 2 } end for end while
CUP’s pseudo-code is presented in Alg.
The modifications CUP made to SAC are marked in red.
Additional implementation details are deferred to Appendix A.4.
Experiments
We evaluate on Meta-World (Yu et al., 2020), a popular reinforcement learning benchmark composedof multiple robot manipulation tasks.
These tasks are both correlated (performed by the same Sawyerrobot arm) and distinct (interacting with different objects and having different reward functions),and serve as a proper evaluation benchmark for policy reuse.
The source policies are achieved bytraining on three representative tasks: Reach, Push, and Pick-Place.
We choose several complex tasksas target tasks, including Hammer, Peg-Insert-Side, Push-Wall, Pick-Place-Wall, Push-Back, and
Shelf-Place.
Among these target tasks, Hammer and Peg-Insert-Side require interacting with objectsunseen in the source tasks.
In Push-Wall and Pick-Place-Wall, there is a wall between the objectand the goal.
In Push-Back, the goal distribution is different from Push.
In Shelf-Place, the robot isrequired to put a block on a shelf, and the shelf is unseen in the source tasks.
Video demonstrationsof these tasks are available at https://meta-world.github.io/ .
Similar to the settings in Yanget al.
(2020a), in our experiments the goal position is randomly reset at the start of every episode.
4.1 Transfer Performance on Meta-World
We compare against several representative baseline algorithms, including HAAR (Li et al., 2019),
PTF (Yang et al., 2020b), MULTIPOLAR (Barekatain et al., 2021), and MAMBA (Cheng et al.,2020).
Among these algorithms, HAAR and PTF learn hierarchical high-level policies over sourcepolicies.
MAMBA aggregates source policies’ V functions to form a baseline function, and performspolicy improvement over the baseline function.
MULTIPOLAR learns a weighted sum of sourcepolicies’ action probabilities, and learns an additional network to predict residuals.
We also compareagainst the original SAC algorithm.
All the results are averaged over six random seeds.
As shownin Figure 2, CUP is the only algorithm that achieves efficient transfer on all six tasks, significantlyoutperforming the original SAC algorithm.
HAAR has a jump-start performance on Push-Wall and
Pick-Pick-Wall, but fails to further improve due to optimization non-stationarity induced by jointlytraining high-level and low-level policies.
MULTIPOLAR achieves comparable performance in
Push-Wall and Peg-Insert-Side, but fails in more complex tasks.
This is because that in complex tasks, learning the residual network can be difficult.
PTF and MAMBA both fail in all six tasks, overlappingeach other at the bottom of the figures.
PTF fails as its hierarchical policy only gets updated when theagent chooses similar actions to one of the source policies, which is quite rare when the source andtarget tasks are distinct.
MAMBA fails as estimating all source policies’ V functions accurately issampling inefficient.
Algorithm performance evaluated by success rate is deferred to Appendix A.5.
4.2 Analyzing the Guidance Policy
This subsection provides visualizations of CUP’s source policy selection.
Fig.
3 shows the percentagesof each source policy being selected throughout training in Push-Wall.
At early stages of training,the source policies are selected more frequently as they have positive expected advantages, whichmeans that they can be used to improve the current target policy.
As training proceeds and the targetpolicy becomes better, the source policies are selected less frequently.
Among these three sourcepolicies, Push is chosen more frequently than the other two source policies, as it is more related to thetarget task.
Figure 4 presents the source policies’ expected advantages over an episode at convergencein Pick-Place-Wall.
The Push source policy and Reach source policy almost always have negativeexpected advantages, which implies that these two source policies can hardly improve the currenttarget policy anymore.
Meanwhile, the Pick-Place source policy has expected advantages close tozero after 100 environment steps, which implies that the Pick-Place source policy is close to the targetpolicy at these steps.
4.3 Ablation Study
This subsection evaluates CUP’s sensitivity to hyper-parameter settings and the number of sourcepolicies.
We also evaluate CUP’s robustness to deal with random source policies, which do notprovide meaningful candidate actions for solving target tasks.
4.3.1 Hyper-Parameter Sensitivity
For all the experiments in Section 4.1, we use the same set of hyper-parameters, which indicates that
CUP is generally applicable to a wide range of tasks without particular fine-tuning.
CUP introduces only two additional hyper-parameters to the underlying SAC algorithm, and we further test CUP’ssensitivity to the additional hyper-parameters on Push-Wall.
As shown in Fig.
5(a), CUP is generallyrobust to the choice of hyper-parameters and achieves stable performance.
4.3.2 Number of Source Policies
We evaluate CUP as well as baseline algorithms on a larger source policy set.
We add three policiesto the original source policy set, which solve three simple tasks including Drawer-Close, Push-Wall,and Coffee-Button.
This forms a source policy set composed of six policies.
As shown in Fig.
CUP is still the only algorithm that solves all the six target tasks efficiently.
MULTIPOLAR suffersfrom a decrease in performance, which indicates that learning the weighted sum of source policies’actions becomes more difficult as the number of source policies grows.
The rest of the baselinealgorithms have similar performance to those using three source policies.
Fig.
7 provides a moredirect comparison of CUP’s performance with different number of source policies.
CUP is able toutilize the additional source policies to further improve its performance, especially on Pick-Place-Walland Peg-Insert-Side.
4.3.3 Interference of Random Source Policies
In order to evaluate the efficiency of CUP’s critic-guided source policy aggregation, we add randompolicies to the set of source policies.
As shown in Fig.
5(b), adding up to 3 random source policies
[Figure]
does not affect CUP’s performance.
This indicates that CUP can efficiently choose which sourcepolicy to follow even if there exist many source policies that are not meaningful.
Related Work
Policy reuse.
A series of works on policy reuse utilize source policies for exploration in value-basedalgorithms (Fernández & Veloso, 2006; Li & Zhang, 2018; Gimelfarb et al., 2021), but they are notapplicable to policy gradient methods due to the off-policyness problem (Fujimoto et al., 2019).
ACTeach (Kurenkov et al., 2020) mitigates this problem by improving the actor over behavior policy’svalue estimation, but still fails in more complex tasks.
One branch of methods train hierarchicalhigh-level policies over source policies.
CAPS (Li et al., 2018) guarantees the optimality of the hierarchical policies by adding primitive skills to the low-level policy set, but is inapplicable to
MDPs with continuous action spaces.
HAAR (Li et al., 2019) fine-tunes low-level policies to ensureoptimality, but joint training of high-level and low-level policies induce optimization non-stationarity(Pateria et al., 2021).
PTF (Yang et al., 2020b) trains a hierarchical policy, which is imitated by thetarget policy.
However, the hierarchical policy only gets updated when the target policy choosessimilar actions to one of the source policies, so PTF fails in complex tasks with large action spaces.
Another branch of works aggregate source policies via their Q functions or V functions on the targettask.
Barreto et al.
(2017) and Barreto et al.
(2018) focus on the situation where source tasks andtarget tasks share the same dynamics, and aggregate source policies by choosing the policy thathas the largest Q at each state.
They use successor features to mitigate the heavy computation costbrought by estimating Q functions for all source policies.
MAMBA (Cheng et al., 2020) forms abaseline function by aggregating source policies’ V functions, and guides policy search by improvingthe policy over the baseline function.
Finally, MULTIPOLAR (Barekatain et al., 2021) learns aweighted sum over source policies’ actions, and learns an auxiliary network to predict residualsaround the aggregated actions.
MULTIPOLAR is computationally expensive, as it requires queryingall the source policies at every sampling step.
Our proposed method, CUP, focuses on the settingof learning continuous-action MDPs with actor-critic methods.
CUP is both computationally andsampling efficient, as it does not require training any additional components.
Policy regularization.
Adding regularization to policy optimization is a common approach toinduce prior knowledge into policy learning.
Distral (Teh et al., 2017) achieves inter-task transfer byimitating an average policy distilled from policies of related tasks.
In offline RL, policy regularizationserves as a common technique to keep the policy close to the behavior policy used to collect thedataset (Wu et al., 2019; Nair et al., 2020; Fujimoto & Gu, 2021).
CUP uses policy regularization asa means to provide additional guidance to policy search with the guidance policy.
Conclusion
In this study, we address the problem of reusing source policies without training any additionalcomponents.
By utilizing the critic as a natural evaluation of source policies, we propose CUP, anefficient policy reuse algorithm without training any additional components.
CUP is conceptuallysimple, easy to implement, and has theoretical guarantees.
Empirical results demonstrate that CUPachieves efficient transfer on a wide range of tasks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 2.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix A.6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section3.1.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.
and Appendix A.2.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplemental materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A.4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
(b) Did you mention the license of the assets?
[Yes] See the supplemental materials.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A.1 Proof for Theorem 1
Proof.
As | (cid:101) Q π ttar ( s, a ) − Q π ttar ( s, a ) | ≤ ϵ for all s ∈ S , a ∈ A , we have that for all s i ∈ S ,the difference between the true value function V π ttar and the approximated value function (cid:101)
V π ttar isbounded:
[Equation]
As π ttar ( ·| s ) is contained in Π st , with (cid:102) π tg defined in Eq.
(7), it is obvious that for alls ∈ S , E
a ∼ (cid:102) π tg ( ·| s )
[Equation]
=(cid:101)
V π ttar ( s ) .
Then for all s i ∈ S ,
[Equation]
A.2 Proof of Theorem
Proof.
According to Pinsker’s inequality (Fedotov et al., 2003), D KL ( π t +1 tar ( ·| s ) || (cid:102)
π tg ( ·| s ))
≥1 2ln2 ||
π t +1 tar ( ·| s ) − (cid:102)
π tg ( ·| s ) || 21 , where || · || 1 is the L1 norm.
So we have that for all s ∈ S ,
||
π t +1 tar ( ·| s ) − (cid:102)
π tg ( ·| s ) || 1 ≤ √ 2 ln 2 δ .
According to the Performance Difference Lemma (Kakade &
Langford, 2002), we have that for all s ∈ S :
[Equation]
where µ (cid:102)
π tg s ( s ′ ) =
(1 − γ ) (cid:80) ∞
t =0 γ t p ( s t = s ′ | s 0 = s, (cid:102)
π tg ) 1 is the normalized discounted stateoccupancy distribution.
Note that
[Equation] [Equation]
[Equation] [Equation]
Eventually, we have V π t +1 tar ( s ) ≥ V
[Equation]
A.3 Critic-Guided Source Policy Aggregation under “Hard” Value Functions
In this section we override the notation Q , V to represent “hard” value functions, and override the notation EA to represent the expected advantage , which is defined as EA π j ( s, π i ) =
396 E
[Equation]
Then Theorem 1 and Theorem 2 can be extended as below.
Theorem 3 Let (cid:101)
Q π ttar be an approximation of Q π ttar such that
[Equation]
Define
[Equation]
Then,
[Equation] [Equation]
Theorem
If
[Equation]
then
[Equation]
where (cid:101) R max = max s,a | r ( s, a ) | is the largest possible absolute value of the reward.
Theorem 3 and Theorem 4 implies that CUP can still guarantee policy improvement under hard
Bellman updates.
Proofs are given below.
Proof for Theorem 3.
As | (cid:101) Q π t tar ( s, a ) − Q π ttar
( s, a ) | ≤ ϵ for all s ∈ S , a ∈ A , we have that forall s i ∈ S , the difference between the true value function V π ttar and the approximated value function(cid:101)
V π ttar is bounded:
[Equation]
As π ttar ( ·| s ) is contained in Π st , with π tg defined in Eq.
(17), it is obvious that for all s ∈ S ,
[Equation]
) (cid:104) (cid:101) Q π ttar ( s, a ) (cid:105) =
(cid:101)
V π ttar ( s ) .
Then for all s i ∈ S ,
[Equation] [Equation]
∞
[Equation]
Proof for Theorem 4.
According to Pinsker’s inequality (Fedotov et al., 2003),
D KL ( π
[Equation]
+1 tar ( ·| s ) ||
[Equation]
π tg ( ·| s )) ≥ 1 2ln2 ||
π t +1 tar ( ·| s ) − (cid:102)
π tg ( ·| s ) || 21 , where || · || 1 is the L1 norm.
So we have that for all s ∈ S ,
||
π t +1 tar ( ·| s ) − (cid:102)
π tg ( ·| s ) || 1 ≤ √ 2 ln 2 δ .
According to the Performance
Difference Lemma (Kakade & Langford, 2002), we have that for all s ∈ S :
[Equation] [Equation]
Note that | Q π ( s, a ) | = | E π [ (cid:80) ∞ i =
γ t r ( s t
+ i , a t + i ) | s t = s, a t = a ] | ≤ E π
[ (cid:80) ∞ t =
γ t (cid:101) R max ] = (cid:101) R max 1 − γ .
Eventually, we have V π t +1 tar ( s ) ≥ V (cid:102)
π tg ( s ) −
[Equation]
2ln
[Equation]
− 2 ϵ 1 − γ .
A.4 Additional Implementation Details
To improve
CUP’s computation efficiency, we store the source polices’ output{ π 1 ( ·| s ) , π 2 ( ·| s ) , ..., π n ( ·| s ) } in the replay buffer.
As we can query source policies withbatches of states, and each state in the buffer only need to be queried for once, CUP is computationally efficient.
Empirically, CUP only takes about 30% more wall-clock time than SAC to run thesame number of environment steps.
All experiments are run on GeForce GTX 2080 GPUs.
Thepolicy regularization is added after 0.5M environment environment steps to achieve more stablelearning.
SAC utilizes two Q functions to mitigate the overestimation error.
When CUP forms the guidancepolicy, we use the max value of the two target Q functions to estimate the expected advantage, whichcontributes to bolder exploration.
Using target networks contributes to more stable training.
For all the CUP experiments, we use the same set of hyper-parameters.
The learning rate is 3e-4.β 1 = 30 , and β 2 = 3 e − 3 .
As for HAAR, we fix the source policies, and train a high-level policy as well as an additionallow-level policy with HAAR’s auxiliary rewards.
A.5 Success Rate Evaluation
Fig.
8 and Fig.9 shows performance evaluated by success rates.
The performance is consistent withperformance evaluated by cumulative return.
A.6 Broader Social Impact
We believe policy reuse serves as a promising way to transfer knowledge among AI agents.
Thisability will enable AI agents to master new skills efficiently.
However, we are also aware of possiblenegative social impacts, such as plagiarizing other AI products by querying and reusing their policies.
A.7 Additional Ablation Study on Hyper-Parameters
We evaluate CUP on a wider range of hyper-parameters.
As shown in Fig.
10, CUP achieves stableperformance even if β 1 and β 2 are more than three times as large as their default values.
A.8 Hyper-Parameter Details
All hyper-parameters used in our experiments are listed in Table 1.
We use the same set of hyperparameters for all six tasks.
We also use the same set of hyper-parameters for both CUP and the SACbaseline.
Most hyper-parameters are adopted from Sodhani et al.
Hyper-Parameter
Hyper-Parameter Values batch size non-linearity
ReLU actor/critic network structure fully connected networks, three fully connected layers with 400 units policy initialization standard Gaussian exploration parameters run a uniform exploration policy 50k steps learning rates for all networks 3e- # of samples / # of train steps per iteration 10 env steps / 1 training step optimizer adam
Episode length (horizon) beta for all optimizers (0.9, 0.999) discount 0.
reward scale 1.
temperature learned # of environment steps before adding KL regularization 500k
A.9 Guidance Policy Analysis for All Tasks
This subsection provides analyses of guidance policies on all six tasks, as shown in Fig.
11 and Fig.
Results demonstrate that the Pick-Place source policy is the most useful in Shelf-Place-V2, HammerV2, and Pick-Place-Wall-V2, while the Push source policy is the most useful in Push-Back-V2 and
Push-Wall-V2.
In Peg-Insert-Side-V2, both source policies are of similar usefulness.
A.10 CUP’s Ability to Use Additional Source Policies
In Fig.
6 the improvement brought by additional source policies is mild, because the original threesource policies have already provided sufficient support for policy reuse, as demonstrated in Fig.13.
To evaluate CUP’s ability to utilize additional source policies, we design another two sets ofsource policies.
Set 1 consists of three source policies that solve Reach-V2, Peg-Insert-Side-V2, and
Hammer-V2, while Set 2 adds source policies trained on Push-Back-V2, Pick-Place-Wall-V2, and
Shelf-Place-V2 to Set 1.
As Set 1 is less related to our target task Push-Wall-V2, CUP must utilizethe additional source policies in Set 2 to improve its performance.
As demonstrated in Fig.
14, CUPcan efficiently take advantage of the additional source policies to achieve efficient learning.
A.11 CUP’s Ability to Ignore Unsuitable Source Policies
To investigate CUP’s ability to ignore unsuitable source policies, we design two source policy sets.
The first set consists of three random policies that are useless for the target task, and the second setadds the Reach policy to the first set.
As demonstrated in Fig.
15, when none of the source policiesare useful, CUP performs similarly to the original SAC, and its sample efficiency is almost unaffected by the useless source policies.
When there exists a useful source policy, CUP can efficiently utilize itto improve performance, even if there are many useless source policies
A.12 CUP’s Source Policy Selection on the Source Task
To further investigate CUP’s formation of the guidance policy, we train CUP on one of the sourcetasks, Push-V2.
As shown in Fig.
16, the corresponding source policy Push is selected frequently.
After the target policy converges, CUP selects the target policy and the Push policy for roughly thesame frequency, as they can both solve the task.
A.13 Additional Experiments
We evaluate CUP on Bin-Picking-V2 and Stick-Pull-V2, two tasks less related to the source policies.
As demonstrated in Fig.
17, in this harder setting, CUP’s performance improvement over SAC issmaller.
To investigate this, we provide an analysis on CUP’s source policy selection.
As shown in
Fig.
18, the smaller improvement is because that source policies are less related to the target tasks, asthey are selected less frequently.
A.14 Limitation and Future Work
CUP assumes that all source policies and the target policy share the same state and action spaces,which limits CUP’s application to more general scenarios.
One possible future direction is to takeinspiration from previous works that map the state and action spaces of an MDP to another MDP with similar high-level structure (Wan et al., 2020; Zhang et al., 2020;
Heng et al., 2022; van der Polet al., 2020b,a).
A.15 Analyzing Non-Stationarity in HRL Methods
To further analyze the advantages of CUP and demonstrate the non-stationarity problem of HRLmethods, we illustrate the percentages of each low-level policy being selected by HAAR’s highlevel policy.
HAAR’s low-level policy set consists of the three source policies and an additionaltrainable low-level policy, which is expected to be selected at states where no source policies giveuseful actions.
As demonstrated in Fig.
19(a) and Fig.
19(b), HAAR’s low-level policy selectionsuffers from a large variance over different random seeds, and oscillates over time.
This is becausethat as the low-level policy keeps changing, the high-level transition becomes non-stationary andleads to unstable learning.
In comparison, as shown in Fig.
19(c) and Fig.
19(d), CUP’s sourcepolicy selection is much more stable and achieves superior performance, as it selects source policiesaccording to expected advantages instead of high-level policies, and avoids the non-stationarityproblem.
[Figure]