Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. To further advance the technology in this area, in this paper we propose a novel framework to incorporate human prior knowledge in the training of DRL agents . Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios , i.e., the unprotected left turn and roundabout , are designedalong with human expert demonstrations . The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60\% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse driving behaviors with human-like features demonstrated by the human expert. We also demonstrate that the imitative expert policy with deep ensemble-based uncertainty estimation can lead to better performance, especially in a more difficult task . As a consequence, the proposed method has shown its great potential to facilitate the applications of DRL-enabled human-like autonomous driving in practice .
Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions . Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios ( unprotected left turn and roundabout ) are designed . The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60\% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse driving behaviors as demonstrated by the human expert. We also demonstrate that the imitative expert policy with deep ensemble-based uncertainty estimation can lead to better performance, especially in a more difficult task .
R
To further advance the technology in
However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this,
R
area, in this paper we propose
paper proposes
R
the training of DRL agents
DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions
R
, i.e., the
(
R
, are designedalong with human expert demonstrations
) are designed
R
with human-like features
as
D
. As a consequence, the proposed method has shown its great potential to facilitate the applications of DRL-enabled human-like autonomous driving in practice