  Recently many algorithms were devised for reinforcement learning (RL) with
function approximation. While they have clear algorithmic distinctions, they
also have many implementation differences that are algorithm-agnostic and
sometimes subtle. Such mixing of algorithmic novelty and implementation
craftsmanship makes rigorous analyses of the sources of performance
improvements difficult. In this work, we focus on a series of inference-based
actor-critic algorithms -- MPO, AWR, and SAC -- to decouple their algorithmic
innovations and implementation decisions. We present unified derivations
through a single control-as-inference objective, where we can categorize each
algorithm as based on either Expectation-Maximization (EM) or direct
Kullback-Leibler (KL) divergence minimization and treat the rest of
specifications as implementation details. We performed extensive ablation
studies, and identified substantial performance drops whenever implementation
details are mismatched for algorithmic choices. These results show which
implementation details are co-adapted and co-evolved with algorithms, and which
are transferable across algorithms: as examples, we identified that tanh policy
and network sizes are highly adapted to algorithmic types, while layer
normalization and ELU are critical for MPO's performances but also transfer to
noticeable gains in SAC. We hope our work can inspire future work to further
demystify sources of performance improvements across multiple algorithms and
allow researchers to build on one another's both algorithmic and
implementational innovations.
