O CCUPY & S PECIFY : I NVESTIGATIONS INTO A M AXIMUM C REDIT A SSIGNMENT O CCUPANCY O BJECTIVE FOR D ATA - EFFICIENT R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review A BSTRACT
The capability to widely sample the state and action spaces is a key ingredient toward building effective reinforcement learning algorithms.
The trade-off between exploration and exploitation generally requires the use of a data model, from which novelty bonuses are estimated and used to bias the return toward wider exploration.
Surprisingly, little is known about the optimization objective followed when novelty (or entropy) bonuses are considered.
Following the “probability matching” principle, we interpret here returns (cumulative rewards) as set points that ﬁxate the occupancy of the state space, that is the frequency at which the different states are expected to be visited during trials.
The circular dependence of the rewards sampling on the occupancy/policy makes it difﬁcult to evaluate.
We provide here a variational formulation for the matching objective, named MaCAO (Maximal Credit Assignment Occupancy) that interprets rewards as a log-likelihood on occupancy, that operate anticausally from the effects toward the causes.
It is, broadly speaking, an estimation of the contribution of a state toward reaching a (future) goal.
It is constructed so as to provide better convergence guaranties, with a complementary term serving as a regularizer, that, in principle, may reduce the greediness.
In the absence of an explicit objective occupancy, a uniform prior is used, making the regularizer consistent with a MaxEnt (Maximum Entropy) objective on states.
Optimizing the entropy on states in known to be more tricky than optimizing the entropy on actions, because of an external sampling through the (unknown) environment, that prevents the propagation of a gradient.
In our practical implementations, the MaxEnt regularizer is interpreted as a TD-error rather than a reward, making it possible to deﬁne an update in both the discrete and continuous cases.
It is implemented on an actor-critic offpolicy setup with a replay
buffer, using gradient descent on a multi-layered neural network, and shown to provide signiﬁcant increase in the sampling efﬁcacy, that reﬂects in a reduced training time and higher returns on a set of classical motor learning benchmarks, in both the dense and the sparse rewards cases.
P ROBLEM STATEMENT
Learning in the real world implies dealing with very large, potentially unlimited environments, over which the data to collect is seemingly inﬁnite.
Efﬁcient exploration is thus one of the key aspects of open-ended learning Santucci et al. (2020), when no ﬁnal model of the environment can feasibly be expected to be engineered or trained.
On the one side, having access to unlimited data is very beneﬁcial for the training of complex multi-layered perceptrons, for they are known to rely on large datasets to improve their performance.
On the other side, the circular dependence between the learning algorithm and the data on which it operates renders the learning very tricky, at high risk of data overﬁtting and trapping in local optima.
The open-ended learning problem is generally addressed through the lens of the reinforcement learning framework (Sutton et al., 1998), where rewards are collected during the interaction, and the selection of action is ﬁt so as to maximize the total number of positive rewards, and prevent the encounter of negative ones.
Fitting behaviour to rewards is however at the risk of ignoring important data from the rest of the environment, where putatively more rewarding regions may be neglected.
The agreement of reward-seeking (that is exploitation) with data collecting (that is exploration), is still one of the fundamental issues of modern artiﬁcial intelligence.
An important effort has recently been put on reframing the reinforcement learning setup into a more general probabilistic inference framework, allowing to link rewards seeking and data modelling under a single perspective (Furmston & Barber, 2010; Levine, 2018;
Haarnoja et al., 2018; Abdolmaleki et al., 2018; Fellows et al., 2019).
This greater focus over the data collection problem is linked to an important set of training algorithms, that contain some forms of exploration bonuses, including “curiosity” drives (Schmidhuber et al., 2009; Pathak et al., 2017), intrinsic rewards (Oudeyer et al., 2007) and pseudo-counts (Bellemare et al., 2016; Tang et al., 2017).
However, at the difference of the classic optimization on rewards alone, where the Bellman optimum is well deﬁned, there is still no consensus about the objective followed when optimizing both on rewards and data collection under a variational perspective (Eysenbach & Levine, 2019).
The data collection problem is effectively shadowed by the reward maximization objective, under which it is still considered as a incidental component.
An important body of work has recently been devoted to addressing the data collection problem as such, with the notable design of the MaxEnt algorithm (Hazan et al., 2019),
State Marginal Matching (Lee et al., 2019) and E3D (Dauc´e, 2020), that aim at ﬁtting the distribution of the states encountered to a uniform distribution, in the absence of deﬁnite rewards.
This is here referred as a MaxEnt-on-state principle (or MaxEnt to be short), not to be confounded with the MaxEnt-on-actions principle implemented in the soft actor critic (Haarnoja et al., 2018) for instance.
Following a MaxEnt objective means optimizing the policy so as the states visited are maximally variable, ideally uniformly visiting all possible states.
We develop in the following a possible extension of the MaxEnt principle, that brings a considerable simpliﬁcation in the expression of the evidence lower bound (ELBO) with regards with the existing literature (Furmston & Barber, 2010; Abdolmaleki et al., 2018; Fellows et al., 2019). In contrast to pure MaxEnt, our approach provides a way to combine the MaxEnt objective with a reward maximization objective, under a variational inference perspective.
An intriguing property of the resulting ELBO formula is that the future states (the ones that are visited after the current observation) play the role of a model for the current data, participating in the elaboration of the returns collected under the current policy.
This gives ways toward optimizing the policy with respect to the distribution of the data, and provides a principled justiﬁcation to the use of intrinsic rewards in the design of reinforcement learning algorithms.
P RINCIPLES 2. P ROBABILITY MATCHING RL
We assume an agent acting in a fully observable environment.
The state of the environment is provided by an observation s ∈ S , with S the set of all possible states.
The agent can act on the environment through its actuators.
Such a motor command is described by a ∈ A , with A the set of all motor commands.
In the following, the capital letters S and A will reﬂect random variables on S and A , and the lower cases s and a will either reﬂect observations or random draw realizations.
The decision of which action to choose relies on a policy , that maps the current observation to the action space, generally expressed in a conditional probabilistic form π ( a | s ) .
A reinforcement learning problem consists in ﬁnding a policy π ∗ that maximizes a certain objective function, without knowing the physical or mechanical properties of the environment.
It is supposed here, for simplicity, that the dynamics of the environment is Markovian (no hidden states).
Moreover, the environment is providing an auxiliary signal called the reward.
Sending an action to the environment makes it possible to access to a new state s (cid:48) , and to obtain a reward r ∈ R .
A classic objective in learning is to maximize the global return, generally described as a discounted sum of future rewards over all possible trajectories.
Let us now denote by s t the state visited at time t and τ ( s t ) =
( s t +1 , ..., s t + T , ... )
a certain pathway that is visited after observing s t .
During this visit, a certain number of rewards can be collected, and R ( τ ) is the (discounted) return obtained over τ , i.e.
R ( τ ) =
(cid:80) t γ t r t , with γ ∈ [0 , 1[ a discounting factor that sums up the rewards up to an “horizon” of the order of 1 1 − γ .
This said, the dynamic programming objective (Bellman, 1966), is the result of π ∗ = max Π E s ∼ p
( S 0 )
,τ ∼ p π ( τ | s ) R ( τ ) , with p ( S 0 ) the distribution of initial states, and Π the set of all conditional policies.
When a state transition model p ( S (cid:48) | s, a ) is provided, the unique solution is given by the dynamic programming recurrent equation in the discrete case (Bellman, 1966).
On the contrary, a large panel of reinforcement learning techniques allow to approach the solution in the model-free setup, assuming an effective sampling of all state-action pairs (Sutton et al., 1998).
We are here interested in a different class of objective function, that rely on ﬁtting the rewards toward probabilities of state occupancies .
A reward should indicate in which proportion the different states (and actions) should be visited (and selected) during trials (with the idea that the states providing high return should be visited more often than the ones providing low returns).
Solving the reinforcement learning problem then means to match the external cue to an actual distribution of visit over states and actions, where a differential in rewards only indicates a difference in the number of visits, allowing to seek rewards in a ﬂexible way (so it is also referred as to “soft” reinforcement learning (Haarnoja et al., 2018)).
This idea stems back from empirical observations on human and animal behaviors, and was coined the “matching law” in the operant conditioning literature (Herrnstein, 1961; Eysenbach & Levine, 2019).
2. S TATE OCCUPANCY AND CONDITIONAL STATE OCCUPANCY
Matching rewards to probabilities can be done in many different ways.
We frame here the probability matching reinforcement learning problem into a state occupancy matching problem .
It relies on the use of an occupancy distribution, that is a density of state visit under a certain policy.
Importantly, it ignores the time order at which the different states are visited, still conserving some aspects of causality between states in the form of conditional probabilities as we see later.
Dating back from
Dayan (1993), an occupancy distribution is a distribution on states, designed so as to match with the distribution measured over the trajectories of the MDP.
Following the deﬁnitions of (Puterman, 2014; Ho & Ermon, 2016; Hazan et al., 2019), a gammaabsorbing state occupancy of a Markov Decision process (with a policy π ) is the (discounted) density of visit of the states — or (state, action) pairs — of the environment when starting from the initial distribution p ( S 0 ) .
It is deﬁned, as:
[Equation]
so that any policy π settled on an MDP deﬁnes an occupancy on the states of that MDP.
It comes that, inversely, any valid (state, action) occupancy (meaning that this occupancy is effectively feasible in a given agent/environment setup), deﬁnes a unique corresponding policy:
[Equation]
that is a softmax (stochastic) conditional policy over the states.
Following the same reasoning, let ρ π ( S + | s... )
the conditional occupancy be deﬁned recursively.
Let
T π ( s ) the set of trajectories starting from s :
[Equation]
The triple dots (...) are intended to help distinguish the one-step distribution p π ( S (cid:48) | s ) from the longterm distribution ρ π ( S + | s... ) .
This conditional distribution provides a description of the “future” of s , that is the distribution of states that will most probably follow s .
It can be seen as an instance of the “successor” representation of states initially proposed by Dayan (1993).
Those future states will generally be noted s + , with the ’+’ exponent meaning the state being measured “further away in time”.
Those deﬁnitions provide a way toward interpreting rewards as occupancy templates, allowing to implement the “matching law” in a principled way. The mapping of rewards toward probabilities relies on using exponentiated returns in the parameters of a stochastic policy, such as in the softmax (or Boltzmann) decision rule case.
Let π ( a | s ) = exp βQ ( s,a ) K ( s ) with K ( s ) = (cid:80) a exp βQ
( s, a ) , with β the “inverse temperature”, and the state-action value Q ( s, a ) representing the total return estimated at ( s, a ) . Let τ = ( s 0 , s 1 , ..., s t , ... ) a certain trajectory observed on the MDP under the policy π .
The set of all possible trajectories is noted T , p π ( τ ) is a measure over the trajectories for a certain policy π , and ρ π is the corresponding occupancy on states . Consider for instance the series of rewards encountered when following τ . It comes that:
[Equation]
so that ∀ t , r ( s t ,a t ) 1 − γ is interpreted as an estimator of V ( s 0 ) . Next, for any t , it comes that ∀ t (cid:48) > t , r ( s t (cid:48) ,a t (cid:48) ) 1 − γ is an estimator of the state-action value Q ( s t , a t ) , i.e:
[Equation]
In that setup, the rewards are interpreted as value samples.
This means, in short, that each future reward r ( s + , a + ) takes the role of a “model” for the total return Q ( s t , a t ) .
The models are weighted according to the conditional occupancy ρ π ( S + , A + | s, a... ) , that takes the role of the “mixture”.
Then, noting that log π ( a | s ) = βQ
( s, a ) − K ( s ) , we deﬁne :
[Equation]
said the “extended” return composed of the return estimator plus a virtual baseline.
Then, because the policy and the occupancy are exchangeable from eq.(2), each reward collected after ( s, a ) may also take the role of a “model” for the occupancy, sampled from the conditional occupancy, i.e.:
[Equation]
2. D ENSITY MATCHING OPTIMIZATION
Assuming π a current policy, ρ π a corresponding occupancy, and taking (cid:104) (cid:103) log ρ ( s, a ) (cid:105)
A + ,S + ,π as a shorthand for
[Equation]
the sampling-based optimization writes:
[Equation]
which is a cross-entropy objective that aims at ﬁtting ρ π ( S, A ) with (cid:104) (cid:101) ρ ( S, A ) (cid:105) .
The optimization can be done, for instance, by optimizing a current policy π by stochastic gradient ascent on the objective, which conducts to maximizing the return, under a softmax policy, like in classic policy gradient.
However, we can make a step further by trying to estimate how far is (cid:104) (cid:103) log ρ ( s, a ) (cid:105) from the optimum log ρ ∗ ( s, a ) .
Interestingly, for any distribution on the future states q ( S + , A + ) , the following inequality holds:
[Equation]
providing a variational (log-) evidence lower bound (ELBO) interpretation of the maximization on β ¯ R ( s, s + , a + ) .
At the convergence of the gradient ascent, the distribution q is expected to match the posterior ρ ∗ , and the inequality would becomes an equality, i.e.
[Equation]
This formula (7) is not directly usable in optimization, because the optimal posterior ρ ∗ is not speciﬁed, but provides new hints into interpreting the current occupancy at the light of its future effects.
The actual optimization is indeed done on ¯ R ( s, s + , a + ) solely, irrespective of the divergence bias.
This bias is expected to fade away with the progress of the training, making the sampling of the rewards more and more accurate at estimating the parameters of the policy/occupancy.
This however, is not guaranteed, and the optimization on the (pseudo) cross entropy is at risk of keeping a high divergence throughout the optimization, hindering the convergence toward the optimum.
This illustrates a more general problem that is the lack of efﬁcacy in sampling the data (the posterior occupancy), a more robust upper bound guarantee would be preferred, even at the risk of a lesser ﬁnal optimality with regard to the Bellman optimum.
2. M AXIMAL C REDIT A SSIGNMENT O CCUPANCY
We now introduce the main insight of our Maximal Credit Assignment Occupancy (MaCAO) model.
It appears that the formulas (5) and (6) are not entirely satisfactory: the average over the rewards translates into an average over log-density templates, that actually depend on ( s + , a + ) , though it is not explicit in the formula.
Expressing this dependency is possible by reshaping the ELBO formula (7) like :
[Equation]
This introduces a new conditional distribution, namely ρ ( S, A | ...s + , a + ) , that is the frequency at which ( s, a ) may precede ( s + , a + ) in the iteration of the dynamics.
This distribution expresses an “anti-causal” relationship between the future states and the current observations, that is the exact measure of how much ( s, a ) is instrumental in reaching ( s + , a + ) .
This is also said the “credit assignment” in the reinforcement learning literature (Sutton, 1988;
Harutyunyan et al., 2019).
Here ρ ∗ ( s, a | ...s + , a + ) represents the target credit assignment.
From the Bayesian perspective, it is interpreted as a log-likelihood of the “data” ( s, a ) , given the “model” ( s + , a + ) , that is a way to say that ( s + , a + ) exerts a control on ( s, a ) .
This conducts to reconsider the rewards as log-likelihood templates rather than occupancy templates.
It comes from lemma 4 (see appendix)) that: E s + ,a + ∼ q ( S + ,A + ) log ρ ∗ ( s, a | ...s + , a + ) ≥ E s [Equation]
[Equation]
S + ,A + ) ¯ R ( s, s + , a + ) , so that:
[Equation]
This new objective provides a variational Bayesian perspective on the density matching optimization, with q (the conditional occupancy) taking the role of the variational distribution.
This is equivalent to augmenting the return with a supplementary divergence term on the conditional occupancy.
Then, matching log ρ ∗ ( s, a ) with the returns becomes identical to maximize the ELBO (8) from variational inference.
The loss is composed of two complementary terms, a ﬁrst term being the occupancy matching on cumulative rewards (that is consistent with the softmax optimization), while the second term is the explicit density matching of a posterior with a prior.
The prior takes the role of a supervision signal, that aims at putting a constraint on the conditional occupancy (that represents the exploration pattern).
By construction of the loss, the right term is made to shape the posterior occupancy, that conducts the evaluation of the return.
This is formally analog to the case of Bayesian inference where the prior serves as a regularizer that tends to counteract the overﬁtting of the data.
This sort of regularizer is highly expected in reinforcement learning that is known to be prone to overﬁtting.
The analogy with Bayesian inference suggests for instance to consider a Gaussian or a uniform distribution.
This has important consequences though.
Taking a prior that is not the target occupancy strongly modiﬁes the interpretation of the loss, that is now composed of two concurrent terms.
The likelihood part aims at ﬁtting the occupancy with the rewards collected, while the divergence part aims at ﬁtting the posterior occupancy with an arbitrary prior.
This breaks the original symmetry, for the implies to concurrently follow two different objectives.
For instance, the role of regularizer that would be devoted to a uniform prior introduces a bias in the probability matching, for the target occupancy is not anymore the softmax-Bellman optimum, but rather an intermediary occupancy that combines the Bellman optimum and a uniform occupancy.
This is expected to be beneﬁcial for a wider exploration, helping to avoid overﬁtting, but this is at the cost of a relaxed constraint on ﬁtting to the optimum.
This expression of a variational upper bound is reminiscent of the standard variational bounds considered in the reinforcement learning literature (Furmston & Barber, 2010; Abdolmaleki et al., 2018), from which it could be interpreted as an occupancy-oriented variant.
By breaking the temporal chain dependence, however, it surprisingly provides a justiﬁcation for considering an entropy on states in the optimization of the policy.
Indeed, the classic variational optimization operates on chained sequences of observations, for which the state transitions vanish during the optimization of π (Haarnoja et al., 2018).
In our case, ignoring the sequential time order establishes both the future states and the future actions as parameters of the policy, over which it should be optimized.
This allows to address the data efﬁciency through explicitly optimizing on the exploration path.
It moreover provides a room to the maxent-on-states term (Hazan et al., 2019), as a regularization incentive in a more general expression.
This conceptual shift has however an important consequence on the objective followed, for the target occupancy is not anymore equal to the softmax Bellman optimum.
M ETHOD
The main ingredients for an efﬁcient implementation is the access to a wide variety of samples ( s, a, r, s (cid:48) , a (cid:48) ) over which optimization can be carried out on parameterized policy π θ (said the “actor”) and a parameterized action value function Q ψ (said the “critic”).
It is here implicitly assumed that both the actor and the critic consist of multi-layered perceptrons, containing many parameters and organized in layered weights, over which a gradient descent is operated on losses expressed as negative objectives.
Assuming an off-policy approach, we consider a replay buffer containing many samples of states, actions and rewards as observed from interacting with the environment.
In a variational setup, one can assume an alternation between two complementary steps.
A ﬁrst step, said the “estimation” step, consists in evaluating a distribution over the parameters of the model, that is here assumed to be a distribution over future states, identiﬁed as the conditional occupancy under the current policy q π .
This distribution is then exploited in a second step, said the “maximization” step, where the parameters of the policy/Q-function are updated so as to maximize the proximal objective (that is ﬁtting a policy to a distribution of returns obtained from the current occupancy).
We assume here the estimation of q π being obtained from a parametric or non-parametric method over a sufﬁciently large sample of recent states (or state, action pairs), and concentrate on the optimization of the actor under the guidance of the critic.
Knowing q π , and given a sample ( s, a, r, s (cid:48) , a (cid:48) ) , an element of optimization is given by the log-difference log q π ( s (cid:48) , a (cid:48) )
− log ρ ∗ ( s (cid:48) , a (cid:48) )
that is a point estimate of the Kullback-Leibler divergence, that is expected to be minimized during the optimization of the policy.
This term is supposed to be differentiated with regards to π , providing a ﬁrst gradient direction that should contribute to improving the policy toward a wider occupancy on future states.
A second and independent element of optimization is the parameterized action-value mapping Q ( s, a ) , relying on the TD-error construct Sutton (1988), based on approximating the future rewards with the best current estimate at t +1 , deﬁning a gamma-discounted proximal objective value ˜ Q ( s, a ) =
r + γQ ( s (cid:48) , a (cid:48) ) .
The squared difference λ ( ˜ Q ( s, a ) − Q ψ ( s, a )) 2 , with λ a precision hyperparameter, is known as the mean-squared Bellman error (MSBE), providing a second gradient that aims at maximizing the return with regards to the policy parameters.
The concurrence of both gradients contains the necessary elements to combine exploration and exploitation in an principled way.
Building a full parametric model of the occupancy is however a difﬁcult task that should be undertaken with care.
The building of such a probabilistic model is indeed at a non negligible cost of regressing parametric distributions from samples, that inherently contain design choices and a speciﬁc optimization on a set of latent parameters (like in the case, e.g., of auto-encoders).
For the sake of simplicity, we consider here the case of a non-parametric estimator of the occupancy distribution q π .
This could appear counter-intuitive at ﬁrst glance, for the update of the policy is supposed to rely on backpropagating gradients through the estimator.
As a workaround, we provide here a method allowing to directly inject the gradient information in the design of the Q -function.
Assume a parameterize Q -function Q ψ that should undergo a dual optimization under two concurrent objectives.
This implies in short that the value settled in the Q -function may arbitrate between the reward-seeking and the occupancy-seeking objectives, so that maximizing π with respect to Q only may be equivalent to the previous concurrent optimization.
Considering ( s, a, r, s (cid:48) , a (cid:48) )
a sample, a simple way to retain the divergence information is to simply add the log difference information log q π ( s (cid:48) , a (cid:48) )
− log ρ ∗ ( s (cid:48) , a (cid:48) ) to the current Q -function, making it possible to recover the original formula.
This sketch of idea implies the use of two concurrent update rules in the Q -function itself. A ﬁrst term L ref ( s, a, r, s (cid:48) , a (cid:48) ) would be the traditional MSBE loss on rewards. The second loss needs to consider the log-difference itself as an error , for it to be “contained” (so to say) in Q after the update. Consider the implicit reward :
[Equation]
(10) and let ˆ Q ( s, a ) =
r KL ( s, a ) + γQ ψ ( s (cid:48) , a (cid:48) ) .
Then a complementary MSBE loss is:
[Equation]
Require: π θ (actor), Q ψ (critic), B (replay buffer), β , γ , λ (hyperparameters) while number of trials not exceeded do initialize the environment while trial not terminated do observe s choose a ∼ π θ ( A | s ) read s (cid:48) , r store ( s, a, r, s (cid:48) ) in B end while if B is full enough then randomly sample a batch of (next) states { s + , ... } from B .
estimate q with a nonparametric method.
while number of batch updates not exceeded do randomly sample a batch of transitions b
= { ( s, a, r, s (cid:48) ) , ... } from B .
for all ( s, a, r, s (cid:48) )
∈ b do estimate log q ( s (cid:48) ) and set r KL (eq. 10) sample a (cid:48)
∼ π θ ( A | s (cid:48) ) calculate L MaCAO ( s, a, r, s (cid:48) , a (cid:48) )
(eq. 11) end for update the critic Q ψ by gradient descent over all losses.
for all s ∈ b do sample a ∼ π θ ( A | s ) estimate Q ψ ( s, a ) calculate L act = − Q ψ ( s, a ) + 1 β log π θ ( A | s ) (including the reparameterization trick) end for update the actor π θ by gradient descent over all losses.
end while end if end while
So that the ﬁnal loss expression is:
[Equation]
[Equation]
The additional precision hyperparameter λ plays here the role of an extra regularizer, helping to adjust between both terms in case of highly sparse rewards.
The main lines of our implementation are provided in algorithm 1 (see Appendix), that ﬁts the pursuit of the MaCAO objective in an actorcritic setup.
It relies on a wide use replay buffers (Mnih et al., 2013) to regularize the gradient over batches that mix the samples from many different trials.
From this perspective, an important shortcut is our on-the-ﬂy calculation of the log-occupancy, with the help of kernel-based density estimation method (Pedregosa et al., 2011) from an initial sampling of (future) states from the buffer at each start of an update sequence (line 10).
This occupancy sample remains quite limited in number (about 1000) in order to avoid unnecessary computer overload.
R ESULTS
In order to reach state-of-the art efﬁcacy, many algorithmic improvements need to be included in supplement to the baseline algorithm 1 .
This concerns in particular the use of target Q-networks updated at slower pace (Mnih et al., 2013), and the clipped double-Q trick (Fujimoto et al., 2018).
Our implementation is moreover drawn over the “spinning-up” open source framework (Achiam, 2018), allowing for a direct comparison with the state of the art.
We consider here for comparison the soft actor-critic method (SAC) (Haarnoja et al., 2018), proximal policy optimization (Schulman et al., 2017) and TD3 (Fujimoto et al., 2018).
Our method was tested over several benchmark environments, as provided by the “Gym” suite (Brockman et al., 2016).
We concentrate here on the continuous states and actions case, that is the most challenging one with regards to function approximation. [Figure] MaCAOSACPPOTD
The different setups are compared on the basis of the returns collected during training.
This is expressed as average return (that is the total sum of rewards gathered at the end of an episode), the average reward (total rewards collected divided by the episode length) and cumulative rewards (the total sum of rewards collected at a given stage of the training).
The width of the occupancy over the state space is not compared here, for the other frameworks are not designed to optimize it.
The different environments differ in scale, difﬁculty and rewards density.
All continuous problems proposed in the library provide dense rewards, that are a compound of negatively and positively weighted extrinsic informations, like the energy consumption, the speed of the agent or its elevation.
The problems separate in two broad categories.
A ﬁrst class of problems provides only dense rewards.
A second class of problems have, in addition, a supplementary sparse reward taking the form of an “endof-episode” bonus or penalty.
In that case, the dense rewards may (or may not) contain relevant information with regards to the task at hand.
From that prospect, the most unfavorable problem is the Continuous Mountain Car problem (ﬁrst row of ﬁgure 4).
Here the dense rewards only refer to the energy consumption, at the exception of a +100 end-of-episode bonus obtained at the hilltop.
This inevitably conducts baseline algorithms to remain stucked at the bottom of the hill, where the energy consumption is low.
Only our approach, that contains an explicit incentive for widening the occupancy of the state space, has the capability to reach the most rewarding states, ﬁnally providing a policy that solves the task.
The Bipedal Walker (second row of ﬁgure 4) is also a problem that combines dense and sparse rewards.
A negative (-100) reward is undergone when the agent falls down, and a positive (+100) reward is gained when the agent reaches the end of the track.
The continual dense rewards provide an incentive for staying upright and increase the velocity.
This task reveals more tricky to train than expected, and contains enough variability for the agent to develop various gaits and locomotion patterns over the course of learning.
Our approach shows here a clear advantage, that is maybe more obvious when comparing with the SAC.
Like in the Mountain car, the problem is about reaching a ﬁnal (distal) end-of-path objective, from which a strong bonus allows to ”freeze” the behavior in a favorable locomotion pattern.
The (S-shaped) discontinuity in the MaCAO and the SAC learning curves reﬂect the reaching of the distal objective, after which close-by policies are followed.
The difference in the two curves is the time at which it is attained, that is less than 10 5 iterations in the ﬁrst case, and more than 10 6 in the second.
This one order difference illustrates the disadvantage of occupancy-agnostic optimization methods.
The swimmer task (third row) is concerned with the development of a locomotion pattern that is swimming in a liquid medium.
The reward is only the speed at which an eel-like agent manage to swim over the place (that is coordinating segments in a periodic manner).
This tasks contain a local optimum that corresponds to a rower pattern that coordinates the extremal segments, and a global optimum that corresponds to a classic swimming ripple from the head toward the tail.
Despite its apparent simplicity, only an extensive exploration such as the one provided by our approach allows to reach the optimum.
The fourth task, known as the ”ant” aims at controlling the locomotion of a 4-legged agent.
The state space contains a detailed account of joint angle and torque moments plus contact sensors in a 111 dimension observation vector (Schulman et al., 2015), but the control space is more reduced (8 DOFs).
Here again the displacement speed is the main incentive, with a survival bonus, and an energy cost penalty.
All 3 actor critic frameworks (namely MaCAO, SAC and E3D) are here capable to reach a decent locomotion pattern in about 400000 iterations of the dynamics, which can be considered data efﬁcient here.
No clear advantage is found here for our approach.
Last, the humanoid task shows a large number of degrees of freedom, and the unlimited number of possible locomotion patterns often result in strange-looking ﬁnal gaits.
Only the SAC and the MaCAO methods allow here to reach decent locomotion patterns in the limited number of steps considered.
When looking in detail, the light advantage observed for the SAC algorithm on the average episode return is reversed when considering the average return.
This apparent contradiction is explained when looking at the detailed behavior.
Here, the high-speed risky locomotion patterns developed in the MaCAO framework result in a higher number of early failures.
This is not related to a risk-seeking incentive, but is rather explained by a tendency to maintain a high diversity of behavior while pursuing the reward-guided objective, which reveals to be more risky when the balance of the body needs to be maintained over time.
D ISCUSSION
This work participates to a general trend toward the development of data models in reinforcement learning, that provide ways to help the agent toward better exploring the world.
This is known of practical use and has been largely exploited so far in the large family of curiosity-driven and maximum-entropy algorithms.
Our contribution here is to provide a more detailed appraisal of the beneﬁts and putative risks of such a construct.
It is shown here to frame into a larger
Bayesian/variational optimization where the future data plays the role of a model, and where an evidence lower bound is maximized through gradient ascent over the policy parameters.
The general principles exposed point to the importance of an occupancy model that synthesizes the general distribution of the agent’s environmental states over which it can act (deﬁning a virtual “territory”).
Those occupancy models are the subject of frequent updates as the exploration progresses and that new states are undisclosed during the course of the training.
By making an additional uniform prior assumption on the occupancy, the resulting loss expresses a balance between two concurrent tendencies, namely the widening of the occupancy space and the maximization of the rewards, reminding of the classical exploration/exploitation trade-off.
The consequence is a shift in the target occupancy pursued, that relaxes the constraint on ﬁtting the initial Bellman objective.
Both are embodied in a MSBE Loss operating on a single Q-function in our implementation (though this is not necessary the case).
Computer simulations illustrate the beneﬁt of our conceptual developments, both in the case of sparse and dense rewards.
A M ATHEMATICAL COMPLEMENTS A. L OWER BOUNDS
Consider a triplet of random variables ( S, A, Z ) deﬁned on the domains S ×
A × Z and obeying to the mixture probability p ( S, A, Z ) . Les s ∈ S be a ”state”, a ∈ A be an “action” and z ∈ Z be a “successor”. Deﬁne p S ) , p ( A ) , p ( Z ) , p ( S, A ) , p ( S, Z ) and p ( A, Z ) the corresponding marginals and deﬁne any conditional p ( X | Y ) =
p ( X,Y ) p ( Y ) .
Lemma 1.
Take π a probability distribution on A , then ∀ s ∈ S , [Equation] p [Equation]
, a ) −
[Equation]
π ( a )
Proof.
Noting that log p ( s, a )
=
log p ( s ) + log p ( a | s ) :
E a ∼ π ( A ) log p ( s, a ) − log π ( a ) =
log p ( s ) − E a ∼ π ( A ) (log π ( a ) − log p ( a | s )) and E a ∼ π {+( A ) log π ( a ) − log p ( a | s ) ≥ 
 (Kullback-Leibler divergence positivity). Lemma 2. Take q a probability distribution on Z , then ∀ ( s, a ) ∈ S × A , log p ( s, a ) ≥ E z ∼ q ( Z ) log p ( s, a | z ) − log q ( z ) + log p ( z )+}
Proof.
Noting that log p ( s, a | z ) + log p ( z ) = log p ( s, a ) + log p ( z | s, a ) (Bayes Theorem) E z ∼ q ( Z ) log p ( s, a | z ) − log q ( z ) + log p ( z ) = log p ( s, a )
{+− E z ∼ q ( Z ) (log q ( z ) − log p ( z )) and E z ∼ q ( Z ) log q ( z ) − log p ( z ) ≥ 
+}
(Kullback-Leibler divergence positivity).
Lemma 3.
Take π a probability distribution on A and q a probability distribution on Z , then ∀ s ∈ S , [Equation] Proof. From lemmas 1 and 2, use the transitivity of the ≥ operator
Remark: the random variable Z has been used here for readability.
Taking Z = ( S + , A + ) with ( S + , A + ) ∈ S × A and p ( Z ) = p ( S + , A + ) , the previous properties still hold.
A. T HE REWARD CONSTRAINT
Take β ∈ R + , γ ∈ ]0 , 1] , and r : Z (cid:55)→ R a reward function.
Note R ( z ) = r ( z ) 1 − γ . Assume that p ( A | s ) (said the “optimal policy”) is such that : [Equation]
) and deﬁne K ( s ) = log (cid:80) a ∈A exp β E z ∼ p ( Z | s,a ) R ( z ) Lemma 4. E
[Equation] Proof. [Equation] From Bayes rule, we know that: ∀ z :
[Equation] Then: [Equation] Noting that:
[Equation]
it comes that: [Equation] Remark: in the main text, ¯ R ( s, z ) = E
z ∼ p
( Z | s
[Equation]
a ) R ( z ) − 1 β ( K ( s ) − log p ( s )) is said the “extended return”.
Deﬁne now π ( A | s ) , a conditional distribution on actions, said the “current policy”.
Then:
Lemma 5.
[Equation] Proof. Identify from lemma (3) π ( A ) ≡ π ( A | s ) and q ( Z ) ≡ p ( Z | s, a ) .
Then from lemma 4, it comes:
[Equation]
which proves the formula.
The right term of the formula is said the (log)-evidence lower bound (ELBO) in the main text.
It is constructed such that β ¯ R ( s, z ) is interpreted as the log likelihood of ( s, a ) (even if it is, more strictly speaking, a lower bound of the log likelihood).
A. O PTIMIZATION
The ELBO being bounded from above by log p ( s ) , one can now deﬁne a (local) optimization criterion.
Say Π be a family of policies.
Then the optimization criterion is:
[Equation]
This criterion contains by construction two optimization objectives. • A ﬁrst objective is said the reward density-matching objective:
[Equation]
It is constructed so that the log-probability of choosing action a is proportional to the average reward observed at further stages of the dynamics. • A second objective is said the regularization objective:
[Equation]
Taking a uniform prior distribution p ( Z ) imposes for the policy to make the successor Z to ﬁt, on average, a uniform distribution.