  Training deep reinforcement learning agents on environments with multiple
levels / scenes from the same task, has become essential for many applications
aiming to achieve generalization and domain transfer from simulation to the
real world. While such a strategy is helpful with generalization, the use of
multiple scenes significantly increases the variance of samples collected for
policy gradient computations. Current methods, effectively continue to view
this collection of scenes as a single Markov decision process (MDP), and thus
learn a scene-generic value function V(s). However, we argue that the sample
variance for a multi-scene environment is best minimized by treating each scene
as a distinct MDP, and then learning a joint value function V(s,M) dependent on
both state s and MDP M. We further demonstrate that the true joint value
function for a multi-scene environment, follows a multi-modal distribution
which is not captured by traditional CNN / LSTM based critic networks. To this
end, we propose a dynamic value estimation (DVE) technique, which approximates
the true joint value function through a sparse attention mechanism over
multiple value function hypothesis / modes. The resulting agent not only shows
significant improvements in the final reward score across a range of OpenAI
ProcGen environments, but also exhibits enhanced navigation efficiency and
provides an implicit mechanism for unsupervised state-space skill
decomposition.
