  In this work, we propose Behavior-Guided Actor-Critic (BAC), an off-policy
actor-critic deep RL algorithm. BAC mathematically formulates the behavior of
the policy through autoencoders by providing an accurate estimation of how
frequently each state-action pair was visited while taking into consideration
state dynamics that play a crucial role in determining the trajectories
produced by the policy. The agent is encouraged to change its behavior
consistently towards less-visited state-action pairs while attaining good
performance by maximizing the expected discounted sum of rewards, resulting in
an efficient exploration of the environment and good exploitation of all high
reward regions. One prominent aspect of our approach is that it is applicable
to both stochastic and deterministic actors in contrast to maximum entropy deep
reinforcement learning algorithms. Results show considerably better
performances of BAC when compared to several cutting-edge learning algorithms.
