Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic