V ARIATIONAL I NTRINSIC C ONTROL R EVISITED
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we revisit variational intrinsic control (VIC), an unsupervised reinforcement learning method for ﬁnding the largest set of intrinsic options available to an agent.
In the original work by Gregor et al.
(2016), two VIC algorithms were proposed: one that represents the options explicitly, and the other that does it implicitly.
We show that the intrinsic reward used in the latter is subject to bias in stochastic environments, causing convergence to suboptimal solutions.
To correct this behavior and achieve the maximal empowerment, we propose two methods respectively based on the transitional probability model and Gaussian Mixture
Model.
We substantiate our claims through rigorous mathematical derivations and experimental analyses.
I NTRODUCTION
Variational intrinsic control (VIC) proposed by Gregor et al.
(2016) is an unsupervised reinforcement learning algorithm that aims to discover as many intrinsic options as possible, i.e., the policies with a termination condition that meaningfully affect the world.
The main idea of VIC is to maximize the mutual information between the set of options and ﬁnal states, called empowerment .
The maximum empowerment is desirable because it maximizes the information about the ﬁnal states the agent can achieve with the available options.
These options are independent of the extrinsic reward of the environment, so they can be considered as the agent’s universal knowledge about the environment.
The concept of empowerment has been introduced in (Klyubin et al., 2005; Salge et al., 2014) along with methods for measuring it based on Expectation Maximization (Arimoto, 1972; Blahut, 1972).
They deﬁned the option as a sequence of a ﬁxed number of actions.
Yeung (2008) proposed to maximize the empowerment using the Blahut & Arimoto (BA) algorithm, but its complexity increase exponentially with the length of sequence, rendering it impractical for high dimensional and longhorizon options.
Mohamed & Rezende (2015) adopted techniques from deep learning and variational inference (Barber & Agakov, 2003) and successfully applied empowerment maximization for high dimensional and long-horizon control.
However, this method maximizes the empowerment over open-loop options, meaning that the sequence of action is chosen in advance and conducted regardless of the (potentially stochastic) environment dynamics.
Obviously this often limits performance severely as the agent cannot properly react to the environment, and it tends to lead to a signiﬁcant underestimation of empowerment (Gregor et al., 2016).
To overcome this limitation, Gregor et al.
(2016) proposed to use the closed-loop options where the agent considers transited states while it is generating a sequence of actions.
This type of option differs from those in Klyubin et al.
(2005), Salge et al.
and Mohamed & Rezende (2015) in that they use closed-loop options and have a termination condition, instead of a ﬁxed number of actions.
They presented two algorithms: VIC with explicit and implicit options (we will call them explicit and implicit VIC from here on).
Explicit VIC deﬁnes a ﬁxed number of options.
An option is sampled at the beginning of the trajectory, conditioning the policies of an agent until the termination.
In other words, both the state and the sampled option are the input to the policy function of the agent.
One clear limitation of explicit VIC is that it requires the preset number of options.
This does not only apply to explicit VIC, but also to some recent unsupervised learning algorithms that adopt a discrete option or skill with a predeﬁned set (Machado et al., 2017; Eysenbach et al., 2018).
Obviously, preset number of options limits the number of options that agent can learn and hence it limits the maximal level of empowerment.
Moreover, choosing a proper number of options is not straightforward, since the maximum of the objective for a given number of options depends on several unknown environmental factors such as the cardinality of the state space and the transitional model.
To overcome this issue, Gregor et al.
(2016) proposed implicit VIC which deﬁnes the option as the trajectory until the termination (Gregor et al., 2016), i.e., the sequence of states and actions until termination.
There exist multiple trajectories that lead to the same ﬁnal state and implicit
VIC learns to maximize the mutual information between ﬁnal state and trajectory by controlling these options (trajectories).
As a result, the number of options is no longer limited by the preset number.
The unlimited number of options is desirable since it means the possibility of unlimited empowerment to an agent. However, despite the charm of the unlimited number of options, these implicit options induce bias in the intrinsic reward and make implicit VIC fails to achieve maximal empowerment. The impact of bias on implicit VIC becomes larger as the environment becomes more stochastic and it can cause serious degradation in empowerment that limits an agent from learning universal knowledge of the environment. In this paper, we aim to solve the empowerment degradation of implicit VIC under stochastic dynamics.
To recover the maximal empowerment of implicit VIC under stochastic dynamics, we revisit variational intrinsic control as follows:
We show that the intrinsic reward in implicit VIC suffers from the variational bias in stochastic environments, causing convergence to suboptimal solutions (Section 2).
To compensate this bias and achieve the maximal empowerment, we suggest two modiﬁcations of implicit VIC: the environment dynamics modeling with the transitional probability (Section 3) and
Gaussian Mixture Model
(Section 4).
V ARIATIONAL B IAS OF IMPLICIT VIC IN STOCHASTIC ENVIRONMENTS
In this section, we derive the variational bias of implicit VIC under stochastic environment dynamics.
First, we adopt the deﬁnition of termination action and ﬁnal state from Gregor et al.
for a clear explanation.
The termination action terminates the option and yields the ﬁnal state s f = s t independently of the environmental action space.
The objective term in VIC, i.e., the mutual information between option Ω and ﬁnal state s f , can be written as follows:
[Equation]
Since p (Ω | s 0 , s f ) is intractable, VIC (Gregor et al., 2016) derives the variational bound I V B ≤ I and optimizes it instead:
[Equation]
where q (Ω | s 0 , s f ) is the inference model to be trained.
When I V B is maximized, we have q (Ω | s 0 , s f ) =
p (Ω | s 0 , s f ) and achieve the maximum I .
Explicit VIC has an additional action space of options.
An option is sampled at s
and then it conditions the policy π until the termination ( π ( a | s, Ω) ).
Unlike explicit VIC, implicit VIC deﬁnes the option Ω as the trajectory of an agent, i.e., the sequence of states and actions: Ω = ( s 0 , a 0 , s 1 , a 1 , ..., s T − 1 , a T − 1 = a f , s T = s f
= s T − 1 ) .
In this case, the probability of an option can be decomposed as
[Equation]
using Bayes rule with τ t
= ( s 0 , a 0 , ..., s t ) .
Similarly, p (Ω | s 0 , s f ) can be expressed as
[Equation]
Note that s f is replaced by s f | Ω since it is determined by the given Ω in the following way:
[Equation]
This is a key characteristic of implicit VIC and is essential for deriving main results of this paper.
Using (3), (4) and (5), we can rewrite the mutual information (1) as
I
[Equation]
The intrinsic reward of implicit VIC (Gregor et al., 2016) is given by
[Equation]
where q ( a t | τ t , s f ) is inference and p ( a t | τ t ) is policy of an agent.
It can be shown that r I VB Ω comes from the ﬁrst part of (6) (see Appendix A for details).
Under deterministic environment dynamics, the transitional part log p ( s t +1 | τ t , a t , s f | Ω ) /p
( s t +1 | τ t , a t ) is canceled out since both the nominator and denominator are always 1.
However, this is not possible under the stochastic environment dynamics and it yields the variational bias b V IC Ω in the intrinsic reward:
[Equation]
In Section 5, we provide the experimental evidence that this variational bias leads to a suboptimal training.
Even though the original VIC (Gregor et al., 2016) subtracts b ( s 0 ) from r I VB Ω to reduce the variance of learning, it cannot compensate this bias since it also depends on Ω .
In the next section, we analyze the mutual information (1) in more detail under stochastic environment dynamics and deﬁne the variational estimate of (1), I V E , for training.
I MPLICIT VIC WITH TRANSITIONAL PROBABILITY MODEL
In this section, we analyze I (Ω , s f | s 0 ) under stochastic environment dynamics and propose to explicitly model transitional probabilities to ﬁx variational bias.
First, for a given option and ﬁnal state, we deﬁne p π
(Ω | s f , s 0 ) , p ρ (Ω | s f , s 0 ) , p π (Ω | s 0 ) and p π (Ω | s 0 ) as follows:
[Equation]
Note that p (Ω | s f , s 0 ) =
p π (Ω | s f , s 0 ) p
ρ (Ω | s f , s 0 ) is the true distribution of Ω for given s f where p π (Ω | s f , s 0 ) is policy related part and p ρ (Ω | s f , s 0 ) is transitional part of p (Ω
| s f , s 0 ) and so do p (Ω | s 0 ) , p π (Ω | s 0 ) and p ρ (Ω | s 0 ) .
Since the variational bias is induced from transitional probabilities, it is necessary to consider them.
Hence we model (9) as follows:
[Equation]
where π q , ρ q , π p and ρ p are our models to be trained.
Since we know the policy of an agent, we have p π ( a t | τ t ) =
π p ( a t | τ t ) .
For the other probabilities, they are trained based on our algorithms.
Now we can rewrite I (Ω , s f | s 0 ) as below using (9):
[Equation]
Using (10), we deﬁne I V E as follows:
[Equation]
This is an estimate of the mutual information between Ω and s f with transitional models.
Note that I V E is not a variational lower bound on I unlike I V B , hence the derivation of implicit VIC in
Gregor et al.
(2016) is not applicable in this case.
To tackle this problem, we start from absolute difference, | I − I V E | which can be bounded as
[Equation]
Algorithm
Initialize s 0 , η , T train , θ qπ , θ qρ , θ pπ
and θ pρ .
for i train : 1 to T train do
Follow π p ( a t | τ t ) result in Ω =
( s 0 , a 0 , ..., s f ) .
r
I VE Ω
[Equation]
(cid:46)
[Equation] [Equation]
(cid:46) from (15), (*)
Update each parameter:
[Equation]
(cid:46) from (15)
[Equation]
(cid:46) from (14)
[Equation]
(cid:46) from (14), (*)
[Equation]
) (cid:46) from (14), (*) end end for
See Appendix B for the derivation.
This upper bound implies that I V E → I as ( π q (Ω | s f , s 0 ) , ρ q (Ω | s f , s 0 ) , ρ p (Ω | s 0 ) )
→ ( p π (Ω | s f , s 0 ) , p ρ (Ω | s f , s 0 ) , p ρ (Ω | s 0 ) ) for all Ω .
(Note that p π (Ω | s 0 ) is omitted since we know the true value of it.)
In other words, our estimate of the mutual information converges to the true value as our estimates (10) converge to the true distribution (9).
It makes sense that we can estimate the true value of the mutual information if we know the true distribution.
Furthermore, U V E
=
if and only if ( π q (Ω | s f , s 0 ) , ρ q (Ω | s f , s 0 ) , ρ p (Ω | s 0 ))
=
p π (Ω | s f , s 0 ) , p ρ (Ω | s f , s 0 ) , p ρ (Ω | s 0 )) for all Ω since U V E is sum of positively weighted KL divergences.
This means that minimizing U V E will make (10) converge to (9).
If π q , ρ q and π p are parameterized by θ qπ , θ qρ and θ pπ , we can obtain ∇ θ qπ U V E , ∇ θ qρ U V E and ∇ θ pρ U V E using (5):
[Equation]
which can be estimated from sample mean.
Once we have ( π q (Ω | s f , s 0 ) , ρ q (Ω | s f , s 0 ) , ρ p (Ω | s 0 ) )
≈ ( p π (Ω | s f , s 0 ) , p ρ (Ω | s f , s 0 ) , p ρ (Ω | s 0 ) ) for all Ω , we can update the policy to maximize I .
The gradients, ∇ θ pπ I and ∇ θ pπ
I V E can be obtained in the following form using (5):
[Equation]
∇
[Equation]
pπ
[Equation]
where θ pπ is parameter of π p (Ω | s 0 ) and p π (Ω | s 0 ) is replaced by π p (Ω | s 0 ) since we know the true value of policy (see Appendix A for details).
From (15), we can see that ∇ θ pπ I V E → ∇ θ pπ I as ( π q (Ω | s f , s 0 ) , ρ q (Ω | s f , s 0 ) , ρ p (Ω | s 0 ))
→ ( p π (Ω | s f , s 0 ) , p ρ (Ω | s f , s 0 ) , p ρ (Ω | s 0 )) for all Ω .
Note that for deterministic environments, we can omit ρ q (Ω | s f , s 0 ) and ρ p (Ω | s 0 ) since they are alwaysand it satisﬁes
I V E
= I V B of (2) and ∇ θ qπ U V E
= −∇ θ
qπ I V B , which means that maximizing I V B is equivalent to minimizing U V E for θ qπ (i.e., it is equivalent to the original implicit VIC).
Algorithm 1 summarizes the modiﬁed implicit VIC with the transitional probability model.
The additional steps added to the original implicit VIC are marked with ( ∗ ) in Algorithm 1.
Note that Algorithm 1 is not always practically applicable since it is hard to model p ( s t +1 | τ t , a t ) and p ( s t +1 | τ t , a t , s f ) when the cardinality of the state space is unknown.
(We can not set the number of nodes for softmax .)
In our experiment of Algorithm 1, we will assume that we know the cardinality of the state space.
This allows us to model p ρ ( s t +1 | τ t , a t ) and p ρ ( s t +1 | τ t , a t , s f ) using softmax .
In the next section, we propose a practically applicable method that avoids this intractability of the cardinality.
I MPLICIT VIC WITH G AUSSIAN M IXTURE M ODEL
In this section, we propose the alternative method to overcome the limitation of Algorithm 1, modeling the smoothed transitional distributions.
This allows us to use the Gaussian Mixture
Model (GMM) (Pearson, 1894) and other continuous distributional models.
First, we smooth p ( s t +1 | τ t , a t , s f ) and p ( s t +1 | τ t , a t ) into f σ ( x t +1 | τ t , a t , s f ) and f σ ( x t +1 | τ t , a t ) with x t +1 =
s t +1 + z t
+
and z t +1 ∼ N ( 0 , σ 2 I n ) :
[Equation]
where S ( τ t , a t , s f )
= (cid:8) s (cid:48)
| p ( s (cid:48) | τ t , a t , s f ) > 0 (cid:9) , S ( τ t , a t ) =
(cid:8) s (cid:48)
| p ( s (cid:48) | τ t , a t ) > 0 (cid:9) and n is the dimension of the state.
Then, using Gaussian Mixture Model (GMM) (Pearson, 1894), we model (16) as f qσ ( x t +1 | τ t , a t , s f ) and f pσ ( x t +1 | τ
[Equation]
Note that if we set n gmm > max τ t ,a t | S ( τ t , a t ) | , (17) can perfectly ﬁt (16).
Now using (17), we deﬁne I V E σ , the variational estimate with smoothing as follows:
[Equation]
Note that I V E σ is not a variational lower bound on I , hence we can not also apply derivation of implicit VIC in Gregor et al.
The upper bound on | I − I V E σ | can be obtained as follows:
[Equation]
with
[Equation]
(cid:105)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
See Appendix C for the derivation.
This upper bound implies U V E σ, 1 → 0 as f qσ (Ω | s f , s 0 ) /f
pσ (Ω | s 0 )
→ p ρ (Ω | s f , s 0 ) /p
ρ (Ω | s 0 ) and U V E σ, 2 → 0 as π q (Ω | s f , s 0 )
→ p π (Ω | s f , s 0 ) for all Ω .
The correct value of mutual information can be estimated from (18) by minimizing U V E σ,
and U V E σ,
and we can directly minimize U V E σ, 2 using gradient descent as (14) using (5):
[Equation] [Equation]
This can be estimated from the sample mean.
Since U V E σ, 2 = 0 if and only if π q (Ω | s f , s 0 )
=
p π (Ω | s f , s 0 ) for all Ω , this update will make π q (Ω | s f , s 0 ) converge to p π
(Ω | s f , s 0 ) .
Unlike U V E σ, 2 , it is difﬁcult to directly minimize U V E σ, 1 due to the absolute symbol.
Therefore, we instead minimize D KL [ f σ ( ·| τ t , a t , s f ) (cid:107) f qσ ( ·| τ t , a t , s f )] and D KL [ f σ ( ·| τ t , a t ) (cid:107) f pσ ( ·| τ t , a t )] to ﬁt (17) to (16).
If f qσ and f pσ are parameterized by θ qρ and θ pρ , the gradients of KL divergences can be obtained as follows:
[Equation]
Algorithm
Initialize s 0 , η , T train , T smooth , θ qπ , θ qρ , θ pπ
and θ pρ .
for i train : 1 to T do
Follow π p ( a t | τ t ) result in Ω =
( s 0 , a 0 , ..., s f ) .
[Equation]
[Equation]
(cid:46) from (22)
[Equation]
Update each parameter:
[Equation]
(cid:46)
[Equation] [Equation]
(cid:46)
[Equation]
(20) ∆
[Equation]
(cid:46) (*) ∆ θ qρ ←(cid:46)
(*) for i smooth : 1 to T smooth do (cid:46) (*)
Sample ( z
[Equation]
[Equation] [Equation]
, z
[Equation] [Equation]
[Equation] [Equation]
[Equation]
(cid:46)
[Equation] [Equation]
(cid:46) from (21) end for
[Equation]
p ρ /T smooth (cid:46) (*) θ qρ ← θ qρ + ∆ θ qρ /T smooth (cid:46) (*) end end for which can be estimated from the sample mean.
These updates will make ( f qσ (Ω | s f , s 0 ) , f pσ (Ω | s 0 )) converge to ( f σ (Ω | s f , s 0 ) , f σ (Ω | s 0 )) .
As ( f qσ (Ω | s f , s 0 ) , f pσ (Ω | s 0 ))
→ ( f σ (Ω | s f , s 0 ) , f σ (Ω | s 0 )) , it satisﬁes U V E σ, 1 → 0 for ﬁnite T and σ (cid:28) d min where T is the average length of trajectories and d min is the minimal distance between two different states (see appendix D for detail).
Hence for small enough σ and ﬁnite length of trajectories, we can minimize U V E σ, 1 to nearly zero.
Once we have ( π q (Ω | s f , s 0 ) , f qσ (Ω | s f , s 0 ) , f pσ (Ω | s 0 ))
≈ ( p π (Ω | s f , s 0 ) , f σ (Ω | s f , s 0 ) , f σ (Ω | s 0 ))
after the update by (20) and (21), we get I V E σ ≈ I .
Now, we can obtain ∇ θ pπ I V E σ using (5):
[Equation]
Note that ∇ θ pπ
I V E σ → ∇ θ pπ I as ( π q (Ω | s f , s 0 ) , f qσ (Ω | s f , s 0 ) /f pσ (Ω | s 0 ))
→ ( p π (Ω | s f , s 0 ) , p ρ (Ω | s f , s 0 ) /p
ρ (Ω | s 0 )) .
Also, it satisﬁes f σ (Ω | s f , s 0 ) /f
σ (Ω | s 0 ))
≈ p ρ (Ω | s f , s 0 ) /p
ρ (Ω | s 0 )) for ﬁnite T and σ (cid:28) d min (see appendix D for detail).
Hence it satisﬁes ∇ θ pπ
I V E σ → ∇ θ pπ I as ( f qσ (Ω
| s f , s 0 ) , f pσ (Ω | s 0 ) )
→ ( f σ (Ω | s f , s 0 ) , f σ (Ω | s 0 ) ).
However, choosing a small enough σ is not straightforward since d min and T depend on the environment.
Too small σ makes the training of f qσ and f pσ unstable due to its extreme gradient near the means.
Another issue of GMM is the choice of a proper n gmm of (17).
We may choose a very large n gmm for the perfect ﬁt but it makes training hard for its complexity.
We leave the proper choice of σ and n gmm as future works and use empirically chosen values ( σ = 0 .
25 and n gmm = 10 ) in this paper.
We summarize our method in
Algorithm 2.
Additional steps added to the original implicit VIC are marked with ( ∗ ) .
E XPERIMENTS
In this section, we evaluate implicit VIC (Gregor et al., 2016), Algorithm 1 and Algorithm 2.
We use
LSTM (Hochreiter & Schmidhuber, 1997) to encode τ t
=
( s 0 , a 0 , ..., s t ) into a vector.
We conduct experiments on both deterministic and stochastic environments and evaluate results by measuring the mutual information I from samples.
To measure I , we rewrite (1) using (5):
[Equation]
which is maximized when s f is distributed uniformly.
We estimate ˆ I using the distribution of s f from the samples, i.e., ˆ p ( s f | s 0 ) .
We apply exponential moving average (0.99 as a smoothing factor) to an average of 5 repetitions for estimating ˆ p ( s f | s 0 ) .
We manually set T max (maximum length of a trajectory) for each experiment such that the termination action is the only available action at T max th action.
For training GMM of Algorithm 2, transitional models are trained to predict the distribution of ∆ x t +1 = x
t +1 − s t instead of
x t +1 since it is easier to learn for model.
For the training, we have a warm-up phase which trains the base function b ( s 0 ) in Gregor et al.
(2016) and the transitional models.
After the warm-up phase, we update the base function, policy and transitional models simultaneously.
Please see Appendix F.1 for details on the hyper-parameter settings.
We compare the algorithms in deterministic environments.
Note that although (8) is zero for deterministic environments, we still train the transitional models of Algorithm 1 and 2 to show the convergence to the optimum of them.
Fig.
1 shows that all three algorithms rapidly achieve the maximal empowerment.
Fig.
2 shows the training results of implicit VIC and Algorithm 2 in the 25 × 25 grid world with 4 rooms used in Gregor et al.
Both implicit VIC and our Algorithm 2 successfully learn passing narrow doors between rooms.
For more results in Mujoco environment (Todorov et al., 2012), please see appendix E.
[Figure]
We compare the algorithms in stochastic environments.
Please see appendix F.2 for the details on the stochasticity of the environments.
Fig.
3 shows results in simple stochastic environments.
We can see that while implicit VIC converges to sub-optimum, our two algorithms achieve the maximal empowerment.
In Fig.
4, implicit VIC fails to reach far rooms, however, our Algorithm 2 reaches every room in the environment.
From 3 and 4, we can notice that implicit VIC fails to reach far states under the stochastic dynamics.
It happens because the variational bias of implicit VIC is accumulated throughout a trajectory, i.e., it gets larger as the length of the trajectory increases.
Fig.
4 also shows the results of training with external rewards.
The same mixed reward is used as Gregor et al.
(2016), r = r
I + αr E with α = 30 .
For training the random agent, only αr E is used and the entropy loss was applied for exploration.
While the random agent and implicit VIC converge to sub-optimum, our Algorithm 2 achieves the optimal solution.
[Figure]
The sub-optimal solution is reaching the closest room.
The optimal solution is trying to reach the special room (the room with the red box) while it enters the closest normal room as soon as possible when it is impossible to reach there due to the stochastic transition.
C ONCLUSION
In this work, we revisited variational intrinsic control.
We showed that for the VIC with implicit options, the environment stochasticity induces a variational bias in the intrinsic reward, leading to convergence to sub-optimum.
To reduce this bias and achieve maximal empowerment, we proposed to model the environment dynamics using either the transitional probability model or the Gaussian mixture model.
Evaluations on stochastic environments demonstrated the superiority of our methods over the original VIC algorithm with implicit options (Gregor et al., 2016).
A D ERIVATION OF INTRINSIC REWARD FROM MUTUAL INFORMATION
Here we derive the intrinsic reward of VIC by taking the gradient of (1) with respect to the parameter of policy θ .
We omit s 0 for simplicity.
Note that p (Ω) , p ( s f ) and p (Ω , s f ) can be parameterized by θ since they are all determined by policy.
We start by rewriting (1):
[Equation]
Then by taking the gradient with respect to θ , we obtain
[Equation]
Using (cid:88)
[Equation]
and (5) we have
[Equation]
23) where r I
[Equation]
Similarly, we can obtain
[Equation]
where
[Equation]
Using (3) and (4)
[Equation]
we can rewrite r I Ω as
[Equation]
Since p θ ( a t | τ t , s f | Ω ) is intractable
[Equation]
we may replace it with variational inference q φ ( a t | τ t , s f | Ω ) which result in
[Equation]
For deterministic environment [Equation] we have p θ ( s t +1 | τ t , a t , s f | Ω ) =
p ( s t +1 | τ t , a t ) = 1 and both r I Ω and r I VE Ω can be reduced into
[Equation]
B D ERIVATION OF U V E
Here we derive U V E from | I −
I V E | with omitted s 0 for simplicity:
[Equation]
p
[Equation]
Using (5), (9) and (10) [Equation] we obtain
[Equation]
C D ERIVATION OF U V E σ, 1 AND U V E σ,
Here we derive (19).
We also omit s 0 for simplicity here.
We start from (cid:12)(cid:12) I − I V E σ (cid:12)(cid:12) : (cid:12)(cid:12) (cid:12) I −
I V E σ (cid:12)(cid:12) (cid:12) = (cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12) (cid:88) Ω
[Equation]
s f
[Equation]
(cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:88) Ω ,s f p (Ω , s f ) (cid:104) log
p π
(Ω | s f ) p
ρ (Ω | s f ) p qπ (Ω | s f )
f qσ (Ω | s f ) − log p ρ (Ω) f pσ (Ω) (cid:105)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ( ∵ p pπ (Ω)
=
p π (Ω)) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:88) Ω ,s f p (Ω , s f ) (cid:104) log
p ρ (Ω | s f ) f pσ (Ω) p ρ (Ω) f qσ (Ω | s f ) + log p π (Ω | s f ) p ρ (Ω | s f ) p qπ (Ω | s f ) p ρ (Ω | s f ) (cid:105)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) (cid:12) (cid:12) (cid:12) (cid:88) Ω ,s f p (Ω , s f ) log p ρ
(Ω | s f ) f pσ (Ω) p ρ (Ω) f q σ (Ω | s f ) (cid:12)(cid:12)(cid:12) (cid:12) (cid:12) (cid:12) + (cid:12)(cid:12)(cid:12) (cid:12) (cid:12) (cid:12) (cid:88) Ω ,s f p
(Ω , s f ) log p π
(Ω | s f ) p
ρ (Ω |
[Equation]
D D ERIVATION OF THE BOUNDS ON I − I σ
Here we derive the bounds on the estimation error of mutual information with smoothing.
First, we derive the upper bound on f σ ( s t +1 | τ t , a t ) :
f σ ( s t +1 | τ t , a t ) = (cid:88)
[Equation]
Obviously, we have
[Equation]
) which results in:
[Equation]
[Equation]
Similarly, we can obtain the bounds of f σ ( s t +1 | τ t , a t , s f ) as follows:
[Equation]
Combining (24) and (25)
[Equation]
we can obtain
[Equation]
Taking log and using log ( a + b ) ≤ log a + ba for a, b > 0 to (26)
[Equation]
we have
[Equation]
which results in −
[Equation]
Using (9) and (27) we obtain − T Ω
[Equation]
By taking the expectation to each side, we have −
[Equation]
with T max = max Ω T Ω and T = (cid:80) Ω ,s f p (Ω , s f | s 0 ) T Ω .
This implies that the estimation error of mutual information with smoothing converges to 0 for ﬁne T max or T as σ → 0 .
Also, for ﬁnite T and σ (cid:28) d min , it satisﬁes | I − I σ | ≈ 0 .
E A DDITIONAL EXPERIMENT RESULTS
Here we show additional experiment results in the Mujoco environments (Todorov et al., 2012) to show the applicability of our Algorithm 2.
We expect that both implicit VIC and our Algorithmwill show similar results as Fig.
1 and Fig.
2 since they are deterministic environments.
We ﬁxed the length of the trajectory as T = 100 to force the agent to learn long enough trajectories.
Each motor’s action space is quantized to 5 actions.
In both HalfCheetah-v3 and Ant-v3, we can observe that interesting movements are learnt by Algorithm 2.
F E XPERIMENTAL DETAILS
Here we specify experimental details and environment details.
F. H YPER - PARAMETERS
Here we show hyper-parameters used in our experiments.
We used a learning rate of 1 e − 3 for Fig. 1 and Fig. 3 and 1 e − 4 otherwise.
[Table]
F. S TOCHASTIC ENVIRONMENTS DETAIL
Here we specify the transition tables used in this paper.
[Table] [Table]
[Table] [Table]