We propose temporally abstract soft actor-critic ( TASAC ), an off-policy RL algorithm that incorporates closed-loop temporal abstraction into the soft actor-critic (SAC) framework in a simple manner. TASAC adds a second-stage binary policy to choose between the previous action and the action output by an SAC actor. It has two benefits compared to traditional off-policy RL algorithms : persistent exploration and an unbiased multi-step Q operator for TD learning . We demonstrate its advantages over several strong baselines across 5 different categories of 14 continuous control tasks , in terms of both sample efficiency and final performance. Because of its simplicity and generality, TASAC can serve as a drop-in replacement for SAC when temporal abstraction is needed.
We propose temporally abstract actor-critic ( TAAC ), an off-policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework in a simple manner. TAAC adds a second-stage binary policy to choose between the previous action and a new action output by an actor. Crucially, its act-or-repeat decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features : persistent exploration and a new compare-through Q operator for multi-step TD backup . We demonstrate TAAC's advantages over several strong baselines across 5 different categories of 14 continuous control tasks . Code is available at URL
D
soft
R
TASAC
TAAC
D
soft
D
(SAC)
R
TASAC
TAAC
R
the
a new
R
SAC actor. It has two benefits compared to traditional off-policy RL algorithms
actor. Crucially, its act-or-repeat decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features
R
an unbiased multi-step
a new compare-through
R
TD learning
multi-step TD backup
R
its
TAAC's
R
, in terms of both sample efficiency and final performance. Because of its simplicity and generality, TASAC can serve as a drop-in replacement for SAC when temporal abstraction is needed.
. Code is available at URL