  Deep reinforcement learning (DRL) is a promising way to achieve human-like
autonomous driving. To further advance the technology in this area, in this
paper we propose a novel framework to incorporate human prior knowledge in the
training of DRL agents. Our framework consists of three ingredients, namely
expert demonstration, policy derivation, and reinforcement learning. In the
expert demonstration step, a human expert demonstrates their execution of the
task, and their behaviors are stored as state-action pairs. In the policy
derivation step, the imitative expert policy is derived using behavioral
cloning and uncertainty estimation relying on the demonstration data. In the
reinforcement learning step, the imitative expert policy is utilized to guide
the learning of the DRL agent by regularizing the KL divergence between the DRL
agent's policy and the imitative expert policy. To validate the proposed method
in autonomous driving applications, two simulated urban driving scenarios,
i.e., the unprotected left turn and roundabout, are designed along with human
expert demonstrations. The strengths of our proposed method are manifested by
the training results as our method can not only achieve the best performance
but also significantly improve the sample efficiency in comparison with the
baseline algorithms (particularly 60% improvement compared to soft
actor-critic). In testing conditions, the agent trained by our method obtains
the highest success rate and shows diverse driving behaviors with human-like
features demonstrated by the human expert. We also demonstrate that the
imitative expert policy with deep ensemble-based uncertainty estimation can
lead to better performance, especially in a more difficult task. As a
consequence, the proposed method has shown its great potential to facilitate
the applications of DRL-enabled human-like autonomous driving in practice.
