G UARDED P OLICY O PTIMIZATION WITH I MPERFECT O NLINE D EMONSTRATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent or human expert guards the training of a student agent by intervening and providing online demonstrations.
Assuming the teacher policy is optimal, it has the perfect timing and capability to intervene the control of the student agent, providing safety guarantee and exploration guidance.
Nevertheless, in many realworld settings it is expensive or even impossible to obtain a well-performing teacher policy.
In this work we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance.
We instantiate an off-policy Reinforcement
Learning algorithm, termed T eacher- S tudent S hared C ontrol ( TS2C ), which incorporates teacher intervention based on trajectory-based value estimation.
Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and lower-bound safety guarantee without being affected by the teacher’s own performance.
Experiments on autonomous driving simulation show that our method can exploit teacher policies at any performance level and maintain a low training cost.
Moreover, the student policy excels the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments.
I NTRODUCTION
In Reinforcement Learning (RL), the Teacher-Student Framework (TSF) (Zimmer et al., 2014; Kelly et al., 2019) incorporates well-performing neural controllers or human experts as teacher policies in the learning process of autonomous agents.
The teacher policy either provides safety guarantee (Peng et al., 2021) or facilitates efficient exploration (Torrey & Taylor, 2013).
At each step, the teacher chooses to intervene the free exploration of the student according to a specific intervention function.
Online data collected from both the teacher and the student policy will be saved into the replay buffer and exploited with off-policy RL algorithms.
The majority of RL methods in TSF assume the availability of a well-performing teacher policy (Spencer et al., 2020; Torrey & Taylor, 2013), thus the behaviors of student will be largely constrained.
The teacher intervention is triggered when the student acts differently from the teacher (Peng et al., 2021) or when the teacher finds the current state worth exploring (Chisari et al., 2021).
This is similar to imitation learning where the performance of learned agent is significantly affected by the quality of demonstrations (Kumar et al., 2020; Fujimoto et al., 2019).
Thus with current TSF methods if the teacher is incapable of providing high-quality demonstrations, the student will be misguided and its final performance will be upper-bounded by the performance of the teacher.
However, it is time-consuming or impossible to obtain a well-performing teacher in many real-world applications such as object manipulation with robot arms (Yu et al., 2020a) and autonomous driving (Li et al., 2022).
As a result, current TSF methods will behave poorly in these settings.
In reality, the coach of Usain Bolt does not necessarily need to run faster than Usain Bolt.
There could be possible learning schemes where a student can outperform the teacher while retaining a lower-bound safety guarantee from the teacher.
In this work we develop a new guarded policy optimization method called T eacher- S tudent S hared C ontrol ( TS2C ).
It follows the setting of a teacher policy and a learning student policy, but relaxes the requirement of the teacher policy’s high-quality demonstrations.
A new intervention mechanism is designed: Rather than triggering intervention based on the similarity of actions of teacher and student, the intervention is now determined by a trajectory-based value estimator.
The student is allowed to conduct actions that deviate from the teacher’s, as long as their expected return is promising.
By relaxing the intervention criterion from step-wise action similarity to trajectory-based value, the student has the freedom to act differently when the teacher fails to provide correct demonstration and thus has the potential to outperform suboptimal teacher.
We conduct theoretical analysis and show that in previous TSF methods the return of the shared control policy is upper bounded by the performance of the teacher policy.
In contrast,
TS2C is not limited by the suboptimal teacher in upper-bound performance, while still retaining a lower-bound performance and safety guarantee.
Extensive experiments show that under the newly proposed method, the teacher-student shared control can learn efficiently and safely under the different levels of teachers while other TSF algorithms are largely bounded by the teacher’s performance.
Furthermore, the student policies trained under the proposed TS2C substantially outperform baseline methods in terms of higher efficiency and lower cost, supporting our theoretical analyses.
B ACKGROUND 2. P RELIMINARIES
We consider an infinite-horizon Markov decision process (MDP), defined by the tuple M = ⟨S , A , P, R, γ, d 0 ⟩ consisting of a finite state space S , a finite action space A , the state transition probability distribution P : S ×
A × S → [0 , 1] , the reward function R : S ×
A → [ R min , R max ] , the discount factor γ ∈ (0 , 1) and the initial state distribution d 0 : S → [0 , 1] .
Unless otherwise stated, π denotes a stochastic policy π : S ×A → [0 , 1] .
The state-action value and state value functions of π are defined as Q π ( s, a ) =
E s
= s,a
= a,a t ∼ π ( ·| s t ) ,s t +1 ∼ p ( ·| s t ,a t ) [ (cid:80) ∞ t =
γ t R ( s t , a t )] and
V π ( s ) =
E a ∼ π ( ·| s )
Q π ( s, a ) .
The optimal policy is expected to maximize the accumulated return
J ( π ) =
E s ∼ d 0 V π
( s ) .
The Teacher-Student Framework (TSF) models the shared control system as the combination of a teacher policy π t which is pretrained and fixed, and a student policy π s to be learned.
The actual actions applied to the agent are deduced from a mixed policy of π t and π s , where π t starts generating action when intervention happens.
The details of the intervention mechanism are described in
Sec. 3.2.
The goal of TSF is to improve the training efficiency and safety of π s with the involvement of π t .
The discrepancy between π t and π s on state s , termed as policy discrepancy, is the L 1 -norm of output difference: ∥ π t ( ·| s ) − π s ( ·| s ) ∥
= (cid:82) A | π t ( a | s ) − π s ( a | s ) | d a.
We define the discounted state distribution under policy π as d π ( s ) =
(1 − γ ) (cid:80) ∞
t =0 γ t Pr ( s t = s ; π, d 0 ) , where Pr π ( s t = s ; π, d 0 ) is the state visitation probability.
The state distribution discrepancy is defined as the difference in L 1norm of discounted state distributions under two policies:
∥
d π t − d π s ∥ 1 = (cid:82) S |
d π t ( s ) − d π s ( s ) | d s .
2. R ELATED W ORK
Teacher-student Learning The idea of transferring knowledge from a teacher policy to a student policy has been explored in reinforcement learning (Zimmer et al., 2014).
It improves the learning efficiency of the student policy by leveraging pretrained teacher policy, usually done by adding auxiliary loss to encourage the student policy to be close to the teacher policy (Schmitt et al., 2018;
Traor´e et al., 2019).
Though our method follows teacher-student transfer framework, an optimal teacher is not a necessity.
Other attempts to relax the need of well-performing teacher models include student-student transfer (Lin et al., 2017; Lai et al., 2020), in which heterogeneous agents exchange knowledge through mutual regularisation (Zhao & Hospedales, 2021).
During training, agents are fully controlled by either the student (Traor´e et al., 2019;
Schmitt et al., 2018) or the teacher policy (Rusu et al., 2016), while our method follows teacher-student shared control where a mixed policy controls the agent.
Learning from Demonstrations Another way to exploit the teacher policy is to collect static demonstration data from it (Widrow, 1964).
The learning agent will then regard them as optimal transitions to utilize.
If the data is provided without reward signals, agent can learn by imitating the teacher’s policy distribution (Ly & Akhloufi, 2020) or trajectory distribution (Ho & Ermon, 2016;
Xu et al., 2019).
It can also learn a parameterized reward function with inverse reinforcement learn- ing (Abbeel & Ng, 2004).
With additional reward signals, agents can perform Bellman updates pessimistically, as most offline reinforcement learning algorithms do (Levine et al., 2020).
The conservative Bellman update can be performed either by restricting the overestimation of Q-function learning (Fujimoto et al., 2019; Kumar et al., 2020) or by involving model-based uncertainty estimation (Yu et al., 2020b; Chen et al., 2021b).
In contrast to the offline learning from demonstration, in this work we focus on the online deployment of teacher policies with teacher-student shared control and show its superiority in reducing the state distributional shift.
Reinforcement Learning with Shared Control
The shared control in RL enables both the expert and the learning agent to generate online samples in the environment (Ross et al., 2011; Kelly et al., 2019).
The switch between two policies are often determined by the expert, either through the manual intervention of human participators (Abel et al., 2017; Chisari et al., 2021) or by referring to the policy distribution of a parameterized expert (Peng et al., 2021).
(da Silva et al., 2020) proposes to ask for expert advice when the learner’s action has high estimated uncertainty.
The framework is empirically justified and only works for agents with discrete action spaces, while we build our algorithm based on theoretical analysis and investigate continuous action space in this paper.
To exploit samples collected with shared control, (Ross et al., 2011; Kelly et al., 2019) compute behavior cloning loss on samples where the expert policy is in control and discard those generated by the learner.
Other algorithms (Mandlekar et al., 2020; Chisari et al., 2021) assign positive reward labels on expert samples and compute policy gradient loss based on the pseudo reward.
All these methods assume the expert policy to be optimal, while our proposed algorithm can make use of a suboptimal expert policy.
There are also researches focusing on provable safety guarantee with shared control (Peng et al., 2021;
Wagener et al., 2021), while we provide an additional lower-bound guarantee of the accumulated reward for our method.
G UARDED P OLICY O PTIMIZATION WITH O NLINE D EMONSTRATIONS
[Figure]
Fig.
1 shows an overview of our proposed method.
In addition to conventional singleagent RL setting, we include a teacher policy π t in the training loop.
The term “teacher” only indicates that the role of this policy is to help the student training.
No assumption on the optimality of the teacher is needed.
The teacher policy is first used to do warmup rollouts and train a value estimator.
During the training of the student policy, both π s and π t receive current state s from the environment.
They propose actions a s and a t , and then a value-based intervention function T ( s ) determines which action should be taken and applied to the environment.
The student policy is then updated with data collected through such shared control.
We first give a theoretical analysis on the general setting of reinforcement learning with shared control in Sec. 3.1.
We then discuss the properties of different forms of intervention function T in
Sec. 3.2.
Based on these analyses, we propose a new algorithm for teacher-student shared control in
Sec. 3.3.
All the proofs in this section are included in Appendix A.1.
3. A NALYSIS ON S HARED C ONTROL
In reinforcement learning with shared control, the teacher policy and the student policy act together and become a mixed behavior policy π b .
The intervention function T ( s ) determines which policy is in charge.
Let T ( s ) = 1 denotes the teacher policy π t takes control and T ( s )
= 0 means otherwise.
Then π b can be represented as π b ( ·| s )
=
T ( s ) π t ( ·| s ) + (1 − T ( s ))
π s ( ·| s ) .
One issue with shared control is that the student policy π s is trained with samples collected by the behavior policy π b , whose action distribution is not always aligned with π s .
A large state distribution discrepancy between two policies ∥
d π b − d π s ∥ 1 can cause distributional shift and ruin the training.
A similar problem exists in behavior cloning (BC), though in BC no shared control is involved and π s learns from samples all collected by the teacher policy π t .
To analyze the state distribution discrepancy in BC, we utilize a useful lemma from (Achiam et al., 2017).
Lemma 3.1.
The state distribution discrepancy between the teacher policy π t and the student policy π s is bounded by their expected policy discrepancy:
[Equation]
We apply the lemma to the setting of the shared control and derive a bound for ∥ d π b − d π s ∥ 1 .
Theorem 3.2.
For any behavior policy π b deduced by a teacher policy π t , a student policy π s and an intervention function T ( s ) , the state distribution discrepancy between π b and π s is bounded by
[Equation]
where β
= E s ∼ dπb
[ T ( s ) ∥
π t ( ·| s ) − π s ( ·| s ) ∥ 1 ] E s ∼ dπb ∥
π t ( ·| s ) − π s ( ·| s ) ∥∈ [0 , 1] is the expected intervention rate weighed by policy discrepancy.
Both Eq.
1 and Eq.
2 bound the state distribution discrepancy by the difference in per-state policy distributions, but the upper bound with shared control is squeezed by the intervention rate β .
In practical algorithms, β can be minimized to reduce the state distribution discrepancy and thus relieve the performance drop during test time.
Based on Thm.
3.2, we further prove in Appendix A.1 that under the setting of shared control, the accumulated returns of behavior policy J ( π b ) and student policy
J ( π s ) can be similarly related.
The analysis in this section does not assume a certain form of the intervention function T ( s ) .
Our analysis provides the insight on the feasibility and efficiency of all previous algorithms with expert-in-the-loop shared control, including (Kelly et al., 2019; Peng et al., 2021; Chisari et al., 2021).
In the following section, we will examine different forms of intervention functions and investigate their properties and performance bounds, especially with imperfect online demonstrations.
3. L EARNING FROM I MPERFECT D EMONSTRATIONS
A straightforward idea to design the intervention function is to intervene when student acts differently to teacher.
This idea resembles imitation learning (IL) methods where student action is optimized based on behavior cloning loss against teacher action at each step.
We model such process with the action-based intervention function T action ( s ) :
[Equation]
wherein ϵ is a predefined parameter.
A similar intervention function is used in (Peng et al., 2021), where the student’s action is replaced by the teacher’s if it has low probability under the teacher’s policy distribution.
To measure the effectiveness of a certain form of intervention function, we examine the return of the behavior policy J ( π b ) .
With T action ( s ) defined in Eq.
3 we can bound
J ( π b ) with the following theorem.
Theorem 3.3.
With the action-based intervention function T action ( s ) , the return of the behavior policy J ( π b ) is lower and upper bounded by
[Equation]
where H =
E s ∼ d
πb H ( π t ( ·| s )) is the average entropy of the teacher policy during shared control.
The theorem shows that J ( π b ) can be lower bounded by the return of the teacher policy π t and extra term related to the entropy of the teacher policy.
It implies that action-based intervention function
T action is indeed helpful in providing training data with high return.
[Figure]
A drawback of action-based intervention function is the strong assumption on the optimal teacher, which is not always accessible.
If we turn to employ a suboptimal teacher, the behavior policy would be burdened due to the upper bound in
Eq. 4.
We illustrate this phenomenon with the example in Fig.
In the figure a slow vehicle in gray is driving in front of the ego-vehicle in blue, which is under shared control.
The student policy is aggressive and would like to overtake the gray vehicle to reach the destination faster, while the teacher intends to follow the vehicle conservatively.
Therefore, π s and π b will propose different actions in the current state, leading to T action = 1 according to Eq. 3.
The mixed policy with shared control will always choose to follow the front vehicle and the agent can never accomplish a successful overtake.
To empower the student to outperform a suboptimal teacher policy, we investigate a new form of intervention function that encapsulates the long-term value estimation into the decision of intervention, designed as follows:
[Equation]
By using this intervention function, the teacher allows student’s action if the teacher can not perform significantly better than student.
T value no longer expects the student to imitate the teacher policy step-by-step.
Instead, it makes decision on the basis of long-term episodic reward.
Taking trajectories in Fig.
2 again
as examples, if the overtake behavior has high return, the student policy will be preferable to T value .
Then the student control will not be intervened by the conservative teacher.
So with the value-based intervention function, the agent’s exploration ability will not be limited by a suboptimal teacher.
Nevertheless, the lower-bound performance guarantee of the behavior policy π b still holds, shown as follows.
Theorem 3.4.
With the value-based intervention function T value ( s ) defined in Eq.
5, the return of the behavior policy π b is lower bounded by
[Equation]
In safety-critical scenarios, the step-wise training cost c ( s, a ) , i.e., the penalty on the safety violation during training, can be regarded as negative reward.
We define ˆ r ( s, a ) =
r ( s, a ) − ηc
( s, a ) as the combined reward, where η is the weighting hyperparameter.
[Equation]
are similarly defined by substituting r with ˆ r in the original definition.
Then we have the following corollary related to expected cumulative training cost, defined by C ( π ) =
E
[Equation]
Corollary 3.5.
With safety-critical value-based intervention function ˆ T value ( s ) , the expected cumulative training cost of the behavior policy π b is upper bounded by
[Equation]
In Eq.
7 the upper bound of behavior policy’s training cost consists of three terms: the cost of teacher policy, the threshold in intervention ϵ multiplied by coefficients and the superiority of π b over π t in cumulative reward.
The first two terms are similar to those in Eq.
6 and the third term means a tradeoff between training safety and efficient exploration, which can be adjusted by hyperparameter η .
Comparing the lower bound performance guarantee of action-based and value-based intervention function (Eq. 4 and Eq. 6), the performance gap between π b and π t can both be bounded with respect to the threshold for intervention ε and the discount factor γ .
The difference is that the performance gap when using T action is in an order of O ( 1 (1 − γ ) 2 ) while the gap with T value is in an order of O ( 11 − γ ) .
It implies that in theory value-based intervention leads to better lower-bound performance guarantee.
In terms of training safety guarantee, value-based intervention function
T value has better safety guarantee by providing a tighter safety bound with the order of O ( 11 − γ ) , in contrast to O ( 1 (1 − γ ) 2 ) of action-based intervention function (see Theorem 1 in (Peng et al., 2021)).
We show in the Sec.
4.3 that the theoretical advances of T value in training safety and efficiency can both be verified empirically.
3. I MPLEMENTATION
Justified by the aforementioned advantages of the value-based intervention function, we propose a practical algorithm called T eacher- S tudent S hared C ontrol ( TS2C ).
Its workflow is listed in Appendix B.
We first train parametric Q-function with data collected by the teacher policy π t during the warmup period.
Gaussian noise is added to the teacher’s policy distribution to increase the state coverage during warmup.
With limited training data the Q-function may fail to provide accurate estimation when encountering previously unseen states.
We employ the idea of ensembling
Q-networks (Chen et al., 2021a).
A set of ensembled Q networks Q ϕ with the same architecture and different initialization weights are built and trained with the same data.
To learn Q ϕ we follow the standard procedure in (Chen et al., 2021a) and optimize the following loss:
[Equation]
(8) where y
=
E s ′
∼D ,a ′ ∼ π t ( ·| s ′ )
+ N (0 ,σ ) (cid:2) r
+ γ Mean (cid:2) Q ϕ ( s ′ , a ′ ) (cid:3)(cid:3) is the Bellman target and D is the replay buffer storing sequences { ( s, a, r, s ′ ) } .
Teacher will intervene when T value returns 1 or the output variance of ensembled Q-networks surpasses the threshold, which means the agent is exploring unknown regions and requires guidance.
We also use Q ϕ to compute the state-value functions in Eq.
5, leading to the following practical intervention function:
[Equation]
In Eq.
2 it is shown that the distributional shift and the performance gap to oracle can be reduced with smaller β , i.e., less teacher intervention.
Therefore, we minimize the amount of teacher intervention via adding negative reward to the transitions one step before the teacher intervention.
Incorporating intervention minimization, we use the following loss function to update the student’s Q-network parameterized by ψ :
[Equation]
where y ′ =
E s ′ ∼D ,a ′ ∼ π b ( ·| s ′ ) (cid:2) r − λ T TS2C ( s ′ )
+ γQ ψ ( s ′ , a ′ ) − α log π b ( a ′ | s ′ )] is the soft Bellman target with intervention minimization.
λ is the hyperparameter controlling the intervention minimization.
α is the coefficient for maximum-entropy learning updated in the same way as Soft Actor
Critic (SAC) (Haarnoja et al., 2018).
To update the student’s policy network parameterized by θ , we apply the objective used in SAC as:
[Equation] [Equation]
E XPERIMENTS
We conduct experiments to investigate the following questions: (1) Can agents trained with TS2C achieve super-teacher performance with suboptimal teacher policies while outperforming methods with other intervention functions?
(2) Can algorithms with shared control provide safety guarantee and improve training efficiency in complex environments, compared with algorithms without it?
Is TS2C robust in teacher policies trained with different algorithms?
To answer questions (1)(2), we conduct preliminary training with the PPO algorithm (Schulman et al., 2017) and save checkpoints on different timesteps.
Policies in different stages of PPO training are used as teacher policies in
TS2C and other algorithms with shared control.
With regard to question (3), we use agents trained with PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018) and Behavior Cloning as the teacher policies from different sources.
[Figure]
4. E NVIRONMENT S ETUP
We conduct the main experiments on the lightweight driving simulator MetaDrive (Li et al., 2022).
One concern with teacher-student shared control is that the student may simply record teacher’s actions and overfit the training environment.
The MetaDrive driving simulator can generate an unlimited number of scenes with various road network and traffic flow for evaluating the generalizability.
We choose 100 scenes for training and 50 held-out scenes for testing.
Examples of the traffic scenes from MetaDrive are shown in Appendix C. To make a thorough analysis, we also make experiments in the MuJoCo simulator (Todorov et al., 2012).
The results are shown in Appendix D.2.
In MetaDrive, the objective is to drive the ego vehicle to the destination without dangerous behaviors such as crashing into other vehicles.
The reward function consists of the dense reward proportional to the vehicle speed and the driving distance, and the terminal +20 reward when the ego vehicle reaches the destination.
Training cost is increased by 1 when the ego vehicle crashes or drives out of the road.
We mainly focus on the average episodic test reward evaluated on the held-out test scenes for a fair comparison between different methods.
Higher test reward means higher generalizable performance in unseen environment.
We also compare the training cost .
Lower training cost reflects a safer training process which is essential if the algorithm is to be applied in real-world environments.
4. B ASELINES AND I MPLEMENTATION D ETAILS
Two sets of algorithms are selected as baselines to compare with.
One includes traditional RL and
IL algorithms without shared control.
By comparing with these methods we can demonstrate how RL with shared control improves the efficiency and safety of training.
Another set contains previous algorithms with expert and learner shared control, including Importance Advising (Torrey & Taylor, 2013) and EGPO (Peng et al., 2021).
The original Importance Advising uses an intervention function based on the range of the Q-function: I ( s ) = max a ∈ A Q D ( s,a ) − min
a ∈
A Q D ( s,a ) , where Q D is the Q-table of the teacher policy.
Such Q-table is not applicable in the Metadrive simulator with continuous state and action spaces, so several approximations are made.
The EGPO algorithm uses an intervention function similar to the action-based intervention function introduced in section 3.2.
More details on the experiment setup are in Appendix C.
[Figure] [Figure]
4. R ESULTS
Super-teacher performance and better safety guarantee The training result with three different levels of teacher policy can be seen in Fig.
The first row shows that the performance of TS2C is not limited by the imperfect teacher policies.
It converges within 200k steps, independent of different performances of the teacher.
EGPO and Importance Advicing is clearly bounded by teacher-medium and teacher-low, performing much worse than TS2C with imperfect teachers.
The second row of
Fig.
3 shows TS2C has lower training cost than both algorithms.
Compared to EGPO and Importance
Advising, the intervention mechanism in TS2C is better-designed and leads to better behaviors.
Better performance with shared control The result of comparing TS2C with baseline methods without shared control can be seen in Fig.
4(a)(b).
We use the teacher policy with medium level of performance to train the student in TS2C. It achieves better performance and lower training cost than baseline algorithms SAC, PPO and BC.
As an outlier, the behavior cloning algorithm does not involve online sampling in the training process, so it has zero training cost.
We also compare
TS2C trained from different levels of the teacher policy with the baseline algorithms without shared control and place the results in Appendix D.1.
Exploiting teacher policies from different sources The result of training with different sources of teacher policy can be seen in Fig.
The figure shows that TS2C can make use of teacher policies from different sources and achieve super-teacher performance consistently.
The Importance
Advising algorithm has bounded performance in accordance with the teacher policy.
EGPO can only utilize the teacher policy trained from PPO and can not be properly guided by the SAC teacher.
This is because the employed SAC teacher is less stochastic than the PPO policy.
Student’s actions have less likelihood in teacher’s action distribution and are less tolerated by the action-based intervention function in EGPO, leading to large intervention rate and consequently large distributional shift.
Our proposed TS2C algorithm does not access teacher internal action distribution and instead intervenes based on the state-action values of teacher policy, so it is robust to the stochasticity of teacher policy.
Ablation Studies We also conduct ablation studies and show the results in Fig.
We find the intervention cost and ensembled value networks are important to the algorithm’s performance, while
[Figure]
different variance thresholds in the intervention function has little influence.
Also, TS2C with actionbased intervention function behaves poorly in accordance with the theoretical analysis in Section 3.2.
4. E FFECTS OF I NTERVENTION F UNCTIONS
[Figure]
We further investigate the intervention behaviors under different intervention functions.
As shown in
Fig. 4(c), the average intervention rate E s
T ( s ) of TS2C drops quickly as soon as the student policy takes control.
The teacher policy only intervenes during a very few states where it can propose actions with higher value than the student.
The intervention rate of EGPO remains high due to the action-based intervention function: the teacher intervenes whenever the student act differently.
We also show different outcomes of action-based and value-based intervention functions with screenshots in the MetaDrive simulator.
In Fig.
7 the ego vehicle happens to drive behind a traffic vehicle which is in an orange trajectory.
With action-based intervention the teacher takes control and keeps following the front vehicle, as shown in the green trajectory.
In contrast, with the value-based intervention the student policy proposes to turn left and overtake the front vehicle as in the blue trajectory.
Such action has higher return and therefore is tolerable by
T TS2C , leading to a better agent trajectory.
C ONCLUSION AND D ISCUSSION
In this work, we conduct theoretic analysis on the shared control mechanism in the Teacher-Student
Framework.
It is found that while RL with shared control has better properties than some imitation learning methods, using an action-based intervention function limits the performance of the student policy.
We then propose TS2C, a value-based shared control learning scheme for online policy optimization with imperfect teachers.
We provide the theoretic guarantees on its exploration ability and safety.
Experiments show that the proposed TS2C method achieves consistent performance independent to the teacher policy being used.
Our work brings progress and potential impact to relevant topics such as active learning, human-in-the-loop methods, and safety-critical applications.
Limitations.
This work assumes the agent can access environmental reward, and defines the intervention function based on value estimations.
This limitation could be handled by considering reward-free settings and employing unsupervised skill discovery in the setting of shared control.
A T HEOREMS IN TS2C A. D ETAILED P ROOF
We start the proof with the restatement of Lem.
3.1 in Sec.
Lemma A.1 (Lemma 4.1 in (Xu et al., 2019)) .
[Equation]
Thm 3.2 can be derived by substituting π and π ′ in Lem A.1 with π b and π s .
Theorem A.2 (Restatement of Thm. 3.2) .
For any behavior policy π b deduced by a teacher policy π t , a student policy π s and a intervention function T ( s ) , the state distribution discrepancy between π b and π s is bounded by policy discrepancy and intervention rate:
[Equation]
(13) where β
=
E s ∼ dπb ∥T ( s )
[ π t ( ·| s ) − π s ( ·| s )]
∥ 1 E s ∼ dπb ∥
π t ( ·| s ) − π s ( ·| s )
∥is the weighted expected intervention rate.
Proof.
[Equation]
Based on Thm.
3.2, we further prove that under the setting of shared control, the performance gap of π s to the optimal policy π ∗ can be bounded by the gap between the teacher policy π t and π ∗ , together with the teacher-student policy difference.
Therefore, training with the trajectory collected with mixed policy π b is to optimize an upper bound of the student’s suboptimality.
The following lemma is helpful in doing this.
Lemma A.3.
[Equation]
Proof.
It is a direct combination of Lemma 4.2 and Lemma 4.3 in (Xu et al., 2019).
Theorem A.4.
For any behavior policy π b consisting of a teacher policy π t , a student policy π s and a intervention function T ( s ) , the suboptimality of the student policy is bounded by
[Equation]
Proof.
[Equation]
Theorem A.5 (Restatement of Thm. 3.3) .
With the action distributional intervention function
T action ( s ) , the return of the behavior policy J ( π b ) is lower and upper bounded by
[Equation]
where R max = max s,a r ( s, a ) is the maximal possible reward
[Equation]
H =
E s ∼ d
πb H ( π t ( ·| s )) is the average entropy of the teacher policy during shared control
[Equation]
Proof.
[Equation]
Therefore, we obtain √
[Equation]
which concludes the proof
[Equation]
To prove Thm.
3.4, we introduce a useful lemma from (Schulman et al., 2015)
[Equation]
Lemma A.6.
[Equation]
Theorem A.7 (Restatement of Thm. 3.4) .
With the value-based intervention function T value ( s ) defined in Eq.
5, the return of the behavior policy π b is lower bounded by
[Equation]
Proof.
[Equation]
which concludes the proof.
Then we prove the corollary related to safety-critical scenarios.
Corollary A.8 (Restatement of Cor. 3.5) .
With safety-critical value-based intervention function ˆ
T value ( s ) , the expected cumulative training cost of the behavior policy π b is upper bounded by
[Equation]
Proof.
We define expected return under policy π with combined reward ˆ r as ˆ J ( π ) , therefore
[Equation]
According to Thm.
A.7, under ˆ T value ( s ) we have
[Equation] [Equation]
Eq.
24 can be immediately proved by combining Eq.
25 and Eq. 26.
D ISCUSSIONS ON THE R ESULTS
Thm.
3.3 has a squared planning horizon 1 (1 − γ ) 2 in the discrepancy term.
This is in accordance with many previous works (Thm. 1 in (Xu et al., 2019), Thm. 4.1 in (Janner et al., 2019) and Thm.in (Schulman et al., 2015)), which include (1 − γ ) 2 in the denominator when it comes to differences of the cumulative return, given the difference in the action distribution.
The order of 11 − γ in Thm. 3. is tight, which dominates the gap in accumulated return.
Nevertheless, the other constant terms, e.g.
R max and the average entropy, can be tighter given some additional assumptions.
We did not derive a tighter bound since the derivation will not be related to the main contribution of this paper, which is the new type of intervention function.
Thm. 3.3 and Thm. 3.4 in their current forms are enough to demonstrate that the value-based intervention function has the advantage of providing more efficient exploration and better safety guarantee compared with action-based intervention function.
B T HE A LGORITHM
The workflow of TS2C during training is show in Alg.
C A DDITIONAL E XPERIMENT D EMONSTRATIONS C. D EMONSTRATIONS OF D RIVING S CENARIOS
The demonstrations of several driving scenarios are shown in Fig.
We provide a demonstrative video showing the agent behavior trained with PPO and our TS2C algorithm in the supplementary materials.
C. I MPLEMENTATION D ETAILS
The MetaDrive simulator (Li et al., 2022) is released under Apache License 2.0.
The experiments are conducted using the RLLib (Liang et al., 2018) and the open-source code of EGPO (Peng et al., 2021).
All algorithms are trained with 200,000 timesteps and 4 different random seeds.
The training time of a single trial is around 6 hours with two CPUs and one GeForce RTX 2080Ti GPU.
With regard to the Importance Advising algorithm, we follow the warmup procedure in TS2C and obtain the teacher Q-value function.
On a certain state, we sample N actions from the teacher’s policy distribution and compute their Q-values.
The intervention happens if the range, i.e., the maximum value minus the minimum value, surpass a certain threshold ε .
The choice of hyperparameters N and ε are shown in the next subsection.
C. H YPER - PARAMETERS
The hyper-parameters used in the experiments are shown in the following tables.
In the TS2C algorithm, larger values of the intervention threshold ε 1 and ε 2 will lead to a more strict intervention criterion and steps with teacher control will be fewer.
In order to control the policy distribution discrepancy, we choose ε 1 and ε 2 to ensure the average intervention rate to be less than 5%.
Nevertheless, different ε 1 in the intervention function has little influence on the algorithm performance, as shown in Fig.
6 of our paper.
The coefficient for intervention minimization λ is simply set to 1.
If used in other environments, it may need some adjustments to fit the reward scale.
The coefficient for maximum entropy learning α is updated during training as in the SAC algorithm.
The number of warmup timesteps is empirically chosen so that the expert value function can be properly trained.
Other parameters follow the setting in EGPO (Peng et al., 2021).
The hyper-parameters of other algorithms follow their original setting.
[Table]
[Table] [Table]
[Table]
D A DDITIONAL E XPERIMENT R ESULTS
We include additional experiment results here
[Table]
D. A DDITIONAL R ESULTS ON M ETA D RIVE
In Fig.
[Table]
we show the results of TS2C trained with various levels of teachers compared with baseline algorithms without shared control.
The value-based intervention proposed by TS2C can utilize all these teacher policies, leading to safer and more efficient training compared to traditional RL algorithms.
Fig.
10 shows the results with different levels of teacher policy.
We show the training cost, training reward and test success rate of TS2C compared with baseline methods with shared control respectively
[Table]
[Figure]
[Figure]
D. A DDITIONAL R ESULTS ON M U J O C O
We conduct an additional experiment in the MuJoCo simulator.
The original code is adapted to fit the Hopper environment.
We train the PPO algorithm and use its checkpoint with average return 1531 as the teacher policy.
When training the TS2C algorithm in the Hopper environment, we follow the warmup-training procedure and keep the hyperparameters unchanged.
The comparative results are shown in Fig.
[Figure]
reward between the TS2C, SAC and the EGPO algorithm.
Our TS2C algorithm achieves highest performance when it converges. As shown in the results, the TS2C algorithm can still achieve best performance in the Hopper environment among baseline algorithms. The SAC algorithm has an unstable performance curve and inferior performance overall. The EGPO algorithm shows a constrained performance and can not outperform the imperfect teacher.