LS-IQ: I MPLICIT R EWARD R EGULARIZATION FOR I NVERSE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent methods for imitation learning directly learn a Q -function using an implicit reward formulation, rather than an explicit reward function.
However, these methods generally require implicit reward regularization for improving stability, mistreating or even neglecting absorbing states.
Previous works show that a squared norm regularization on the implicit reward function is effective, but do not provide a theoretical analysis of the resulting properties of the algorithms.
In this work, we show that using this regularizer under a mixture distribution of the policy and the expert provides a particularly illuminating perspective: the original objective can be understood as squared Bellman error minimization, and the corresponding optimization problem minimizes the χ 2 -Divergence between the expert and the mixture distribution.
This perspective allows us to address instabilities and properly treat absorbing states.
We show that our method, Least Squares Inverse
Q-Learning (LS-IQ), outperforms state-of-the-art algorithms, particularly in environments with absorbing states.
Finally, we propose to use an inverse dynamics model to learn from observations only.
Using this approach, we retain performance in settings where no expert actions are available.
I NTRODUCTION
Inverse Reinforcement Learning (IRL) techniques have been developed to robsutly extract behaviors from expert demonstration and solve the problems of classical Imitation Learning (IL) methods (Ng et al., 1999; Ziebart et al., 2008).
Among the recent methods for IRL, the Adversarial Imitation
Learning (AIL) approach (Ho & Ermon, 2016; Fu et al., 2018; Peng et al., 2021), which casts the optimization over rewards and policies into an adversarial setting, have been proven particularly successful.
These methods, inspired by Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), alternate between learning a discriminator, and improving the agent’s policy w.r.t.
a reward function, computed based on the discriminator’s output.
These explicit reward methods require many interactions with the environment as they learn both a reward and a value function.
Recently, implicit reward methods (Kostrikov et al., 2020; Arenz & Neumann, 2020; Garg et al., 2021) have been proposed.
These methods directly learn the Q -function, significantly accelerating the policy optimization.
Among the implicit reward approaches, the Inverse soft Q-Learning (IQ-Learn) is the current stateof-the-art.
This method modifies the distribution matching objective by including a reward regularization term on the expert distribution, which results in a minimization of the χ 2 -divergence between the policy and the expert distribution.
However, whereas their derivations only considered regularization on the expert’s distribution, their practical implementations have shown that regularizing the reward on both the expert and policy distribution achieves significantly better performance.
The contribution of this paper is twofold: First, we show that the resulting objective when using this regularizer minimizes the χ 2 divergence between the expert and a mixture distribution between the expert and the policy.
We then investigate the effects of regularizing w.r.t.
the mixture distribution on the theoretical properties of IQ-Learn.
This divergence is bounded, and we show that this translates to bounds on the Q -function and reward functions, improving the stability of learning.
Indeed, the resulting objective corresponds to least-squared Bellman error minimization and is closely related to Soft Q-Imitation Learning (SQIL) (Reddy et al., 2020).
Second, we formulate Least Squares Inverse Q-Learning (LS-IQ), a novel IRL algorithm.
By following the theoretical insight coming from the analysis of the χ 2 regularizer, we tackle many sources of instabilities of the IQ-Learn approach: the arbitrariness of the Q -function scales, exploding Qfunctions targets, and reward bias Kostrikov et al.
(2019), i.e., assuming that absorbing states provide the null reward.
We derive the LS-IQ algorithm by exploiting structural properties of the Q -function and heuristics based on expert optimality.
This results in increased performance on many tasks and, in general, more stable learning and less variance in the Q -function estimation.
Finally, we extend the implicit reward methods to the IL from observation setting by training an Inverse-Dynamics
Model (IDM) to predict the expert actions, which are no longer assumed to be available.
Even in this challenging setting, our approach retains performance similar to the original setting.
Related Work.
The vast majority of IRL and IL methods build upon the Maximum Entropy (MaxEnt) IRL framework (Ziebart, 2010).
In particular, Ho & Ermon (2016) introduce Generative
Adversarial Imitation Learning (GAIL), which applies GANs to the IL problem.
While the original method minimizes the Jensen-Shannon divergence to the expert distribution, the approach was extended to general f -divergences
(Ghasemipour et al., 2019), building on the work of Nowozin et al. (2016).
Among the f -divergences, the Pearson χ 2 divergence was shown to improve the training stability for GANs (Mao et al., 2017) and for AIL (Peng et al., 2021).
Kostrikov et al.
(2019) introduces a replay buffer for off-policy updates for policy and discriminator.
The authors also pointed out the problem of reward bias, which is common in many imitation learning methods.
Indeed, AIL methods implicitly assign a null reward to these states, leading to survival or termination biases, depending on the chosen divergence.
Kostrikov et al.
(2020) improves the previous work introducing recent advances from offline policy evaluation (Nachum et al., 2019).
Their method,
ValueDice, uses an inverse Bellman operator that expresses the reward function in terms of its Qfunction, to minimize the reverse Kullback-Leibler divergence to the expert distribution.
Arenz &
Neumann (2020) derive a non-adversarial formulation based on trust-region updates on the policy.
Their method, O-NAIL, uses a standard Soft-Actor Critic (SAC) (Haarnoja et al., 2018) update for policy improvement.
O-NAIL can be understood as an instance of the more general IQ-Learn algorithm
(Garg et al., 2021), which can optimize different divergences depending on an implicit reward regularizer.
Garg et al.
(2021) also show that their algorithm achieves better performance using the χ 2 divergence instead of the reverse KL.
Reddy et al.
(2020) propose a method that uses SAC and assigns fixed binary rewards to the expert and the policy, and show that the resulting objective is related to regularized behavioral cloning.
Swamy et al.
(2021) provide a unifying perspective on many of the aforementioned methods, explicitly showing that GAIL, ValueDice, MaxEnt-IRL, and
SQIL can be viewed as moment matching algorithms.
P RELIMINARIES
Notation.
A Markov Decision Process (MDP) is a tuple ( S , A , P, r, γ, µ 0 ) , where S is the state space,
A is the action space, P : S ×
A × S → R + is the transition kernel, r : S ×
A → R is the reward function, γ is the discount factor, and µ 0 : S → R + is the initial state distribution.
At each step, the agent observes a state s ∈ S from the environment, predicts an action
a ∈
A using the policy π : S ×
A → R + , and transitions with probability P ( s ′ | s, a ) into the next state s ′ ∈ S , where it receives the reward r ( s, a ) .
We define an occupancy measure ρ π ( s, a ) =
π ( a | s ) (cid:80) ∞ t =0 γ t µ πt ( s ) , where µ πt ( s ′ ) =
(cid:82) s,a µ πt ( s ) π ( a | s ) P ( s ′ | s, a ) da ds is the state distribution for t > 0 , with µ π 0 ( s )
= µ 0 ( s ) .
The occupancy measure allows us to denote the expected reward under policy π as E ρ π
[ r ( s, a )] ≜ E [ (cid:80) ∞ t =0 γ t r ( s t , a t )] , where s 0 ∼ µ 0 , a t ∼ π ( .
| s t ) and s t +1 ∼ P ( .
| s t , a t ) for t > 0 .
Furthermore, R S×A =
{ x : S ×
A → R } denotes the set of functions in the state-action space and R denotes the extended real numbers R ∪ { + ∞} .
We refer to the soft value functions as ˜ V ( s ) and ˜ Q ( s, a ) , while we use V ( s ) and Q ( s, a ) to denote the value functions without entropy bonus.
Inverse Reinforcement Learning as an Occupancy Matching Problem.
Given a set of demonstrations consisting of states and actions sampled from an expert policy π E , IRL aims at finding a reward function r ( s, a ) from a family of reward functions R = R S×A assigning high reward to samples from the expert policy π E and low reward to other policies.
We consider the frame- work presented in (Ho & Ermon, 2016), which derives the maximum entropy IRL objective from an occupancy matching problem with an additional convex reward regularizer ψ ρ : R S×A → R :
[Equation]
where Π = R S×A is the space of policies, H ρ ( π ) is the discounted cumulative entropy bonus such that H ρ ( π ) =
E ρ π
[ − log( π ( a | s ))] , and β is a constant controlling the entropy bonus.
Note that the inner optimization is a maximum entropy Reinforcement Learning (RL) objective, for which the optimal policy is given by (Ziebart, 2010):
[Equation]
where Z s = (cid:82) ˆ a exp ˜ Q
( s, ˆ a )
d ˆ a is the partition function and ˜ Q ( s, a ) is the soft action-value function, which is given for a certain reward function by the soft Bellman operator ( ˜ B π ˜ Q )( s, a )
=
r ( s, a ) + γ E s ′ ∼ P
( . | s,a )
˜ V π ( s ′ ) , where ˜ V π ( s ′ ) =
E a ∼ π ( .
| s )
[ ˜ Q ( s, a ) − log π ( a | s )] .
Garg et al.
(2021) transform Equation 1 from reward-policy space to ˜ Q -policy space using the inverse soft Bellman operator ( ˜ T π ˜ Q )( s, a ) =
˜ Q ( s, a ) − γ E s ′ ∼ P ( .
| s,a )
˜ V π ( s ′ ) to get a one-to-one correspondence between the reward and the ˜ Q -function.
This allows to change the objective function L ρ from reward-policy to Q -policy space, from now on denoted as J ρ :
[Equation]
where ˜Ω = R S×A is the space of ˜ Q -functions.
Further, they use Equation 2 to extract the optimal policy π ˜ Q given a ˜ Q -function to drop the inner optimization loop in Equation 1 such that:
[Equation]
Practical Reward Regularization.
Garg et al.
(2021) derive a regularizer enforcing an L2 normpenalty on the reward on state-action pairs from the expert, such that ψ π E ( r
[Equation]
=
[Equation]
E
[Equation]
with c being a regularizer constant.
However, in continuous action spaces, this regularizer causes instabilities.
In practice, Garg et al.
(2021) address this instabilities by using the regularizer to the mixture
[Equation]
where α is typically set to 0 .
It is important to note that this change of regularizer does not allow the direct extraction of the policy from Equation 1 anymore.
Indeed, the regularizer in Equationalso depends on the policy.
Prior work did not address this issue.
In the following sections, we will provide an in-depth analysis of this regularizer, allowing us to address the aforementioned issues and derive the correct policy update.
Before we introduce our method, we use Proposition A.1 in
Appendix A to change the objectives L ρ and J ρ from expectations under occupancy measures to expectations under state-action distributions d π E and d π , from now on denoted as L and J .
L EAST S QUARES I NVERSE Q-L EARNING
In this section, we introduce our proposed imitation learning algorithm, which is based on the occupancy matching problem presented in Equation 1 using the regularizer defined in Equation 5.
We start by giving an interpretation of the resulting objective as a χ 2 divergence between the expert distribution and a mixture distribution of the expert and the policy.
We then show that the regularizer allows us to cast the original objective into a Bellman error minimization problem with fixed binary rewards for the expert and the policy.
A RL problem with fixed rewards is a unique setting, which we can utilize to bound the Q -function target, provide fixed targets for the Q -function on expert states instead of doing bootstrapping, and properly treat absorbing states.
However, these techniques need to be applied on hard Q -functions.
Therefore, we switch from soft action-value functions ˜ Q to hard
Q -functions, by introducing an additional entropy critic.
We also introduce a regularization critic allowing us to recover the true policy update corresponding to the regularizer in Equation 5.
Finally, we propose to use an IDM to solve the imitation learning from observations problem.
3. I NTERPRETATION AS A S TATISTICAL D IVERGENCE
Ho & Ermon (2016) showed that their regularizer results in a Jensen-Shannon divergence minimiza- tion between the expert’s and the policy’s state-action distribution.
Similarily, Garg et al.
(2021) showed the that their regularizer ψ π E ( r ) results in a minimzation of the χ 2 divergence.
However, the regularizer presented in Equation 5 is not investigated yet.
We show that this regularizer results in minimizing a χ 2 divergence between the expert’s state-action distribution and a mixture distribution between the expert and the policy.
Therefore, we start with the objective presented in Equation 1 yet under state-action distributions instead of occupancy measures, and note that strong duality
[Equation]
follows straightforwardly from the minimax theorem (Von Neumann, 1928) as − H ( π ) , − E d π [ r ( s, a )] and ψ ( r ) are convex in π , and − E d π [ r ( s, a )] , E d πE [ r ( s, a )] and ψ ( r ) are concave in r (Ho & Ermon, 2016).
We express the χ 2 divergence between the expert’s distribution and the mixture distribution using its variational form,
[Equation]
where c = 14 is the regularizer constant.
Now, if the optimal reward is in R , the original objective from Equation 1 becomes an entropy-regularized χ 2 divergence minimization problem:
[Equation]
Unlike the χ 2 -divergence between the agent’s and the expert’s distribution, the χ 2 -divergence to the mixture distribution is bounded.
When the expert and the policy distribution do not share much support, optimizing an unbounded divergence can result in unbounded rewards that are difficult to estimate, significantly destabilizing the training.
Note that the χ 2 -divergence between the expert and the mixture is upper bounded by the χ 2 -divergence between the expert and the policy, as shown in
Proposition A.2.
3. A R EINFORCEMENT L EARNING P ERSPECTIVE ON D ISTRIBUTION M ATCHING
In the following, we present a novel perspective on Equation 4 allowing us to better understand the effect of the regularizer.
Indeed, for the regularizer defined in Equation 5, we can interpret this objective as an entropy-regularized least squares problem, as shown by the following proposition:
Proposition 3.1 Let ψ ( r ˜ Q ) =
c E ˜ d (cid:104) r ˜ Q ( s, a ) 2 (cid:105) with ˜ d ( s, a ) =
αd π E ( s, a ) + (1 − α )
d π ( s, a ) , then arg max ˜ Q ∈ ˜Ω J ( ˜ Q, π ˜ Q ) is the solution to the entropy-regularized least squares
minimization problem arg min ˜ Q ∈ ˜Ω L ( ˜ Q, π ˜ Q ) with:
[Equation]
where r max = 12 αc and r min = −
[Equation]
The proof is provided in Appendix A.
The resulting objective in Equation 10 is very similar to the one in the Least Squares
Generative Adversarial Networks (LSGANs) (Mao et al., 2017) setting, where r ˜ Q ( s, a ) can be interpreted as the discriminator, r max can be interpreted as the target for expert samples, and r min can be interpreted as the target for samples under the policy π .
For α = 0 .
5 and c = 1 , resulting in r max = 1 and r min
=
− 1 , Equation 10 differs from the discriminator’s objective in the LSGANs setting only by the entropy term.
Now applying the inverse soft Bellman operator to the implicit reward r ˜ Q ( s, a ) in Equation 10 and rearranging the terms yields
[Equation] [Equation]
where δ 2 is the squared soft Bellman error.
From Equation 12 we can deduce the following: χ 2 -regularized IRL under a mixture can be seen as a RL problem , with fixed rewards r max and r min for the expert and the policy.
This insight allows us to understand the importance of the regularizer constant c : it defines the rewards and the scale of the Q -function.
The resulting objective shows strong relations to the SQIL algorithm, in which also fixed rewards are used.
However, SQIL uses r max = 1 and r min = 0 , which is infeasible in our setting for α < 1 .
While the entropy term appears to be another difference, we note that it does not affect the critic update, where π ˜ Q is fixed.
As in SQIL, the entropy is maximized by extracting the MaxEnt policy using Equation 2.
Stabilizing the training in a fixed reward setting is straightforward: We can have a clean solution to the reward bias problem – c.f., Section 3.4 –, and we can provide fixed Q -target for the expert and clipped Q -function targets for the policy – c.f., Section 3.5 & 3.7 to significantly improve learning stability.
However, to apply these techniques, we must switch from soft to hard action-value functions by introducing an entropy critic.
Additionally, we show how to recover the true policy update corresponding to the regularizer in Equation 5 by introducing a regularization critic.
3. E NTROPY AND R EGULARIZATION C RITIC
We express the ˜ Q -function in terms of a hard Q -function (the critic) and an entropy critic H π ( s, a ) , that is, ˜ Q ( s, a ) =
Q ( s, a ) + H π ( s, a ) , where
[Equation]
This procedure allows us to stay in the MaxEnt formulation while retaining the ability to operate on the hard Q -function.
We replace the soft inverse Bellman operator with the hard optimal inverse
Bellman operator ( T Q )( s, a ) =
Q ( s, a ) − γ E s ′ ∼ P ( .
| s,a ) V ∗ ( s ′ ) , where V ∗ ( s ) = max a Q ( s, a ) .
As mentioned before, the regularizer introduced in Equation 5 incorporates yet another term depend- ing on the policy.
Indeed, the inner optimization problem in Equation 1—the term in the brackets—is not purely the MaxEnt problem anymore, but includes the term − E d π [ r ( s, a ) 2 ] .
To incorporate this term into our action-value function, we learn an additional regularization critic C ( s, a ) , such that
[Equation] [Equation]
) , where
[Equation]
Using Q † , we obtain the exact solution to the inner minimization problem in Equation 2.
In practice, we learn a single critic G π combining H π and C .
We train the latter independently by minimizing the following Bellman error:
[Equation] [Equation]
3. T REATMENT OF A BSORBING S TATES
Another technical aspect neglected by IQ-Learn is the proper treatment of absorbing states.
Garg et al.
(2021) treat absorbing states by adding an indicator ν —where ν
= 1 if s ′ is a terminal state—in front of the discounted value function in the inverse Bellman operator,
[Equation]
This inverse operator is obtained by solving the forward Bellman operator for r ( s, a ) under the assumption that the value of absorbing states is zero.
However, as pointed out by Kostrikov et al.
(2019), such assumption may introduce termination or survival bias; instead, also the value of absorbing states needs to be learned.
Our perspective provides a clear understanding of the effect of the inverse Bellman operator in Equation 16:
When substituting r ˜ Q ( s, a ) in Equation 10 the objective will regress the Q -function of transitions into absorbing states towards r max or r min , respectively.
However, based on Equation 10, the implicit reward of absorbing states should be regressed toward r max or r min .
Instead, we derive our inverse operator form the standard Bellman operator, while exploiting that the value of the absorbing state
s A is independent of the policy π ,
[Equation]
We further exploit that the value of the absorbing state can be computed in closed form as V ( s A ) =
r A 1 − γ , where r A equals r max on expert states and r min on policy states.
Please note that the corresponding forward Bellman operator converges to the same Q -function, despite using the analytic value of absorbing states instead of bootstrapping, as we show in Appendix A.5.
When applying our inverse operator Equation 17 to Equation 10, we correctly regress the Q -function of transitions into absorbing states towards their discounted return.
We show the resulting full objective in Appendix A.4.
We show the effect of our modified operator on the toy task depicted in Figure 1 (top), where the black point mass is spawned in either of the four dark blue squares and has to reach the green area in the middle.
Once the agent enters the red area, the episode terminates.
The expert always takes the shortest path to the green area, never visiting the red area.
The operator proposed by IQ-Learn does not sufficiently penalize the agent for reaching absorbing states, preventing the IQLearn agent from reaching the goal consistently, as can be seen from the orange graph in Figure 1 (bottom).
In contrast, when using our operator T lsiq , the agent solves the task successfully.
3. A N A LTERNATIVE F ORMULATION FOR THE E XPERT R ESIDUAL M INIMIZATION
The first term in Equation 12 defines the squared Bellman error minimization problem on the distri- bution d
[Equation]
Due to bootstrapping, this minimization can become challenging, even for a fixed expert policy, as it does not fix the scale of the Q -function unless the trajectory reaches an absorbing state.
This problem arises particularly on expert data for cyclic tasks, where we generate trajectories up to a fixed horizon.
The lack of a fixed scale increases the variance of the algorithm, affecting the performance negatively.
Therefore, we propose a modified objective, analyzing the minima of Equation 18.
The minimum of this term is achieved when r Q ( s, a ) =
r max ∀ ( s, a ) ∼ d π E .
Thus, the objective of this term is to push the reward, on expert trajectories, towards r max .
If we consider this minimum, each step of in the trajectory of the expert has the following Q -value:
[Equation]
As the objective of our maximization on expert distribution is equivalent to pushing the value of the expert’s states and actions towards Q max , we propose to replace the bootstrapping target with the fixed target Q max resulting in the following new objective:
[Equation]
Note that we skip the terminal state treatment for clarity.
The full objective is shown in Appendix
A.4.
Also, we omit the entropy term as we incorporate the the latter now in H π
( s, a ) .
This new objective incorporates a bias towards expert data.
Therefore, it is not strictly equivalent to the original problem formulation.
However, it updates the Q -function towards the same ideal target, while providing a much simpler and stable optimization landscape.
Empirically, we experienced that this modification, while only justified intuitively, has a very positive impact on the performance of the algorithm.
3. L EARNING FROM O BSERVATIONS
[Figure]
In many real-world tasks we do not have access to expert actions, but only to observations of expert’s behavior (Torabi et al., 2019b).
In this scenario AIL methods, such as GAIfO (Torabi et al., 2019a), can be easily adapted by learning a discriminator only depending on the current and the next state.
Unfortunately, it is not straightforward to apply the same method to implicit rewards algorithms that learn a Qfunction.
The IQ-Learn method (Garg et al., 2021) relies on a simplification of the original objective to perform updates not using expert actions but rather actions sampled from the policy on expert states.
However, this reformulation is not able to achieve good performance on standard benchmarks as shown in our experimental results.
A common practice used in the literature is to train an IDM.
This ap- proach has been previously used in behavioral cloning (Torabi et al., 2018;
Nair et al., 2017) and for reinforcement learning from demon- strations (Guo et al., 2019; Pavse et al., 2020; Radosavovic et al., 2021).
Following the same idea, we generate an observation-only ver- sion of our method by training an IDM online based on policy data and using it for predicting unobserved actions of the expert.
The approach is described visually in Fig.
We modify the objective in Equationas follows
[Equation]
with the dynamics model Γ( s, s ′ ) .
We omit the notation for absorbing states and refer to Appendix
A.4 instead.
Notice that the IDM is only used to evaluate the expert actions, while it is trained by minimizing the following loss:
[Equation]
where the expectation is performed on the state distribution generated by the learner policy π .
While the mismatch between the training distribution and the evaluation distribution could potentially cause problems, our empirical evaluation shows that on the benchmarks we achieve performance similar to the action-aware algorithm.
3. P RACTICAL A LGORITHM
Algorithm
Initialize: Q θ , π ϕ , G ζ and optionally Γ ω for step t in { 1 , . . .
, N } do
Sample mini-batches D π and D π E (opt.)
Predict actions for D π E using Γ ω { a π E 1 , . .
., a π E |D πE | } ← Γ ω ( D π E )
D π E ← D π E ∪ { a π E 1 , . . .
, a π E |D πE | }
Update the Q -function using Eq.
21 θ t +1 ← θ t + κ Q ∇ θ [ J ( θ, D π , D π E )] (opt) Update G -function using Eq.
15 ζ t +1 ← ζ t − κ G ∇ ζ
[ L G ( ζ, D π )]
Update Policy π ϕ using the KL ϕ t +1 ← ϕ t − κ π ∇ ϕ
[ D KL ( πϕ ∥ π ˜ Q )] (opt.)
Update Γ ω using Eq.
22 ω t +1 ← ω t − κ Γ ∇ ω
[ L Γ ( ω, D π )] end for
We now instantiate a practical version of our algorithm in this section.
An overview of our method is shown in
Algorithm 1.
In practice, we use target networks, which was already suggested by the Garg et al.
However, while the objective in Equation 4 lacked intuition about the usage of target networks—without the regularizer the application of target networks would result in a maximization/minimzation of the Q -function—the objective in Equation 12 is equivalent to a reinforcement learning objective, in which target networks are a well-known tool for stabilization.
Further, we exploit our access to the hard Q -function as well as our fixed reward target setting to calculate the maximum and minimum Q-value possible – Q min = r min 1 − γ and Q max = r max 1 − γ – and clip the output of target network to that range.
Note that this also holds for the absorbing states.
In doing so, we ensure that the Q -function is always forced to stay in the desired range, which we often experienced not the case in IQ-Learn.
Target clipping prevents the explosion of the Q-values that can occur due to the use of neural approximators.
This technique allows the algorithm to recover from poor value function estimates and prevents the Q -function from leaving the set of admissible functions for the problem.
Finally, we found that training the policy
[Figure]
on a small fixed expert dataset quickly anneals the entropy bonus of expert trajectories, even if the policy never visits these states and actions.
To address this problem, we clip the entropy bonus on expert states to a running average of the maximum entropy on policy states.
In doing so, we facilitate exploration towards expert regions.
In continuous action spaces, Z s is intractable, which is why we can not directly extract the optimal policy using Equation 2.
As done in previous work (Haarnoja et al., 2018; Garg et al., 2021), we use a parametric policy π θ to approximate π ˜ Q by minimizing the Kullback-Leibler Divergence (KL)
D KL ( π θ ∥ π ˜ Q ) .
In our implementation, we found it unnecessary to use a double-critic update.
This choice reduces the computational and memory requirements of the algorithm, making it comparable to SAC.
Finally, we replace V ∗ ( s ) with V π ( s ) on the policy expectation, as we do not have access to the latter in continuous action spaces.
E XPERIMENTS
We evaluate our method on five MuJoCo environments: Ant-v3, Walker2d-v3, Hopper-v3,
HalfCheetah-v3 and Humanoid-v3.
We select the following baselines: GAIL (Ho & Ermon, 2016),
VAIL (Peng et al., 2019), IQ-Learn (Garg et al., 2021) and SQIL (Reddy et al., 2020).
For a fair comparison we reimplemented all methods in the same framework, MushroomRL (D’Eramo et al., 2021), such that the same libraries and RL agents were used.
We verified that our implementations achieve comparable results to the original implementations by the authors, where we used their hyperparameters, which were already tuned for the same environments that we use.
The original implementation of IQ-Learn tested two different variants of the algorithm, depending on the given environment.
We refer to these variants as IQv0—which uses telescoping (Garg et al., 2021) to evaluate the agent’s expected return in Equation 4—, and IQ—which directly uses Equation 4—and evaluate both variants on all environments.
For our method, we used the same hyperparameters as
IQ-Learn, except for the regularizer coefficient c and the entropy coefficient β which we tuned on each environment.
For the mixing parameter, we only considered equal mixing, i.e., α = 0 .
In our first experiment, we performed ablations on the different design choices of LS-IQ.
We evaluated the following variants: LSIQ-HC uses a (combined) entropy critic and regularization critic,
LSIQ-H only uses the entropy critic and LSIQ does not use any additional critic, similar to IQLearn.
We used ten seeds and five expert trajectories for these experiments.
We also added IQ, IQv and SQIL as baselines and show the learning curves for four of the environments in Figure 3.
The learning curves on the HalfCheetah environment can be found in Appendix B.
It is also interesting to note that IQ-Learn without telescoping, does not perform well on Walker and Hopper, where absorbing states are more likely compared to Ant and HalfCheetah (which always terminates after a fixed number of steps).
Our hypothesis is that the worse performance on Walker and Hopper is caused by reward bias, as absorbing states are not sufficiently penalized.
IQv0 would suffer less from this problem, as it treats all states visited by the agent as initial states which results in stronger reward penalties for these states.
We conducted further ablation studies showing the influence of the proposed techniques including an ablation study on the influence of fixed targets, clipping on the target Q -value, entropy clipping for the expert, as well as the treatment of absorbing states in Appendix
B. Our results show that the additional critics have little effect, while fixing the targets significantly increase the performance.
To further emphasize the importance of our stabilization techniques on
[Figure]
{+more complex tasks, in Appendix B.
+}
, we present another experiment on a locomotion task using a simulated Atlas robot.
In this task, the performance gap w.r.t.
IQ-learn is more pronounced.
For our main experiments we only evaluated LSIQ and LSIQ-H, which achieved the strongest performance on most environments.
We compare to all baselines for four different numbers of expert demonstrations, namely 1 , 5 , 10 and 25 , and always use five seeds.
We performed each experiment with and without action observations.
When actions are not available, we used a state transition discriminator (Torabi et al., 2019a) for GAIL and VAIL, and IDMs (see Section 3.6) LS-IQ.
In contrast,
IQ-Learn just uses actions predicted on expert states by the current policy when no expert actions are available as proposed by the original authors.
In the learning from observation setting, we did not evaluate SQIL, and we omit the plots for IQ, which did not converge in any environment, and focus on only on IQv0.
Figure 4 shows the final expected return over different numbers of demonstrations for four of the environments.
All learning curves, including the HalfCheetah environment, can be found in Appendix B.6 for state-action observations, and in Appendix B.5 for the learning from observation setting.
Our experiments show that LS-IQ achieves on par or better performance compared to all baselines.
In particular in the learning from observation setting, LS-IQ performs very well by achieving similar return compared to the setting where states and actions are observed.
C ONCLUSION
Inspired by the practical implementation of IQ-Learn, we derive an implicit reward distribution matching algorithm using a squared L2 penalty on the mixture distribution of expert and policy.
We show that this regularizer minimizes the χ 2 -divergence to the mixture distribution, and results in modified updates for Q -function and policy.
Our analysis reveals an interesting connection to
SQIL—which was not derived from an adversarial distribution matching objective—and shows that
IQ-Learn suffers from reward bias.
We build on our insights to propose a novel method, LS-IQ, which uses a modified inverse Bellman operator to address reward bias, target clipping, fixed reward targets for policy samples and fixed Q -function targets for expert samples.
We also show that the policy optimization of IQ-Learn is not consistent with regularization on the mixture distribution, and show how this can be addressed by learning an additional critic.
In our experiments, LS-IQ outperforms strong baseline methods, in particular when learning from observations, where we train an IDM for predicting expert actions.
In future work, we will quantify the bias introduced by the fixed Q -function target and why this heuristic is fundamental for stabilizing learning.
We will also analyze the error propagation in the Q -function target and derive theoretical guarantees on the Qfunction approximation error.
A P ROOFS AND D ERIVATIONS
In this section, we present the proofs of this work.
A. F ROM O CCUPANCY M EASURES TO D ISTRIBUTIONS
Based on Proposition A.1, the solution arg max Q ∈ ˜Ω J ρ ( ˜ Q, π ˜ Q ) under occupancy measures equals the solution arg max ˜ Q ∈ ˜Ω J ( ˜ Q, π ˜ Q ) under state-action distributions.
This result allows us to use the following distribution matching problem from now on:
[Equation]
where we introduce the implicit reward r ˜ Q ( s, a )
=
˜ Q ( s, a ) − γ E s ′ ∼ P ( .
| s,a )
[ ˜ V π ( s ′ )] for comprehension, β is a constant controlling the entropy regularization, d π E is the state-action distribution of the expert, and d π is the state-action distribution under the policy.
Proposition A.1 Let max r ∈R min π ∈ Π L ρ ( π, r ) be the dual problem of a regularized occupancy matching optimization problem and L ( π, r ) be the Lagrangian of the regularized distribution matching problem.
Then it holds that L ρ ( π, r ) ∝ L ( π, r ) and J ρ ( ˜ Q, π ˜ Q ) ∝ J ( ˜ Q, π ˜ Q ) .
Proof of Proposition A.1.
Starting from the definition of the occupancy measure of an arbitrary policy π , we compute the normalizing constant as an integral:
[Equation]
Now we compute the state-action distribution as:
[Equation]
Thus, we have:
[Equation]
Using equation 26 into the definition of the objective we obtain:
[Equation]
A derivation similar to equation 27 can be done for the entropy and the regularizer using equation 26.
Substituting the derived formulas into equation 1 and collecting the constant 11 − γ proves the proposition.
■
A. T HE B OUND OF THE χ 2 D IVERGENCE ON M IXTURE D ISTRIBUTIONS
Proposition A.2 Let χ 2 (cid:0)
d π E (cid:13)(cid:13)
αd π E
+ (1 − α )
d π (cid:1) be the Pearson χ 2 -divergence between the distribution d π E and the mixture distribution αd π E
+ (1 − α )
d π , and let α = 0 .
Then it holds that:
[Equation]
Proof of Proposition A.2.
The proof follows straightforwardly from the joint convexity of f - divergences:
[Equation]
where κ ∈ [0 , 1] is the a mixing coefficient.
The χ 2 -divergence is an f -divergence with f ( t ) =
( t − 1) 2 .
Now using the χ 2 -divergence, and let
P = P 1 = P 2 = Q 2 , D = Q 1 , and α = (1 − κ ) :
[Equation]
Setting P =
d π E and D =
d π concludes the proof.
■
A.
F ROM χ 2 - REGULARIZED M AX E NT -IRL TO L EAST -S QUARES R EWARD R EGRESSION
We recall that the entropy-regularized IRL under occupancy measures is given by
[Equation]
We also recall that using soft inverse Bellman operator to do the change of variable yields:
[Equation]
We can now use Proposition A.1 to switch the objective from occupancy measures to distributions
[Equation]
Proof of Proposition
[Equation]
Staring from the objective function in Equation 23, by expanding the expectation and rearranging the terms, we obtain:
[Equation]
Defining r max = 12 αc and r min = − 1 2(1 − α ) c and completing the squares we obtain:
[Equation]
Finally
[Equation]
we obtain the following result:
[Equation]
where K = αc r 2max + (1 − α ) c r 2min = 14 αc + 1 4(1 − α ) c is a fixed constant.
Comparing Equation 10 with Equation 30 results in
[Equation]
Given that an affine transformation (with positive multiplicative constants) preserves the optimum, arg max ˜ Q ∈ ˜Ω ¯ J ( ˜ Q, π ˜ Q ) is the solution of the entropy-regularized least squares objective L ( ˜ Q, π ˜ Q ) .
■
A. F ULL LS-IQ O BJECTIVE WITH T ERMINAL S TATES H ANDLING
Inserting our inverse Bellman operator derived in Section 3.4 into the least-squares objective defined in Equation 10 and rearranging the terms yields the following objective for hard Q -functions:
[Equation]
Now including the fixed target for the expert distribution introduced in Section 3.5 yields:
[Equation]
where Equation 35 is the full LS-IQ objective for our hard Q -function.
For the observations-only setting we predict the expert’s actions using the IDM.
A. C ONVERGENCE OF OUR F ORWARD B ACKUP O PERATOR
As described in Section 3.4, our inverse operator,
[Equation]
is based on the standard Bellman backup operator, except that, instead of bootstrapping, we use the known values for transitions into absorbing state.
We will now show that repeatedly applying the corresponding forward Operator
[Equation]
converges to the Q function.
Our proof is based on the same technique that is commonly used to prove convergence of the standard Bellmen operator, namely by showing that the Q function
Q
[Equation]
s
[Equation]
′ ) is a fixed point of our operator, that is, ( B π lsiq Q π )( s, a ) =
Q π ( s, a ) , and by further showing that our operator is a contraction,
[Equation]
were || .
|| ∞ is the maximum norm; here, we assume a finite states and actions for the sake of simplicity.
Proposition A.3 The Q function of policy π is a fixed point of B π lsiq ,
[Equation]
Proof of Proposition A.3.
The proof follows straightforwardly from the fact that our forward operator performs the same update as the standard Bellman operator, if applied to the actual Q function of the policy, Q π ( s, a ) , since then V ( s A ) :
= r
( s A ) (1 − γ ) =
E a ′ ∼ π ( .
| A ) Q π ( s ′ , a ′ )) .
Thus, (
[Equation]
■
Proposition A.4 The forward operator B π lsiq is a contraction,
[Equation]
Proof of Proposition A.4.
[Equation]
[Table]
[Equation]
48) ■
B A DDITIONAL E XPERIMENT R ESULTS
This section contains additional results that have been omitted in the main paper due to space constraints.
B. A TLAS L OCOMOTION E XPERIMENTS
Figure 5 presents the training results on a locomotion task of a simulated Atlas robot.
As can be seen, our approaches significantly outperform the baselines emphasizing the importance of our techniques for stable training.
[Figure]
B. A BLATION S TUDY : A BSORBING S TATE T REATMENT AND T ARGET Q C LIPPING
Figure 6 presents ablations on the effects of the proposed treatment of absorbing states and the clipping of the Q -function target on an LSIQ agent with bootstrapping. To see the pure effect of the absorbing state treatment and the clipping, we did not include fixed targets.
Note that the fixed target implicitly treats absorbing states of the expert, as it provides the same target for states and actions transitioning towards absorbing states.
We have chosen a LSIQ agent without an entropy and regularization critic.
The experiments are conducted on the Humanoid-v3 task, as the tasks HalfCheetah-v3, Ant-v3, either do not have or have very rare absorbing states and Walker-v3 and Hopper-v3 are too easy to see the effects.
We use a regularizer constant of c = 0 .
5 and a mixing parameter of α = 0 .
5 yielding a maximum reward of r max = 1 2(1 − 0 .
5 = 2 and a minimum reward of r min = − 1 2(1 − 0 .
5 = − 2 .
This yields a maximum Q -value of Q max = r max 1 − γ = 200 and a minimum Q -value of Q min = r min 1 − γ = − 200 .
The rows show the different agent configurations: First, LSIQ with clipping and absorbing state treatment; second, LSIQ with clipping but no absorbing state treatment; and lastly, LSIQ without clipping and no treatment of absorbing states – which is equivalent to SQIL with symmetric reward targets.
For a better understanding of the effect, the plots show the individual seeds of a configuration.
As can be seen, the first LSIQ agent can successfully learn the task and regresses the Q -value of the transitions towards absorbing states to the minimum Q -value of -200.
The second configuration does not treat absorbing states and is not able to learn the task with all seeds.
As can be seen, the average Q -value of absorbing states is between -
and -6.
Taking a closer look at the first two plots in the second row, one can see that those seeds did not learn, whose average Q -value on non-absorbing transitions is close or even below the average Q -values of states and actions yielding to absorbing states.
This strengthens the importance of our terminal state treatment, which pulls Q -values of states and action towards absorbing states to the lowest possible Q -value and, therefore, avoids a termination bias.
Finally, one can see the problem of exploding Q -value in the last LSIQ configuration.
This is evident by the scale of the abscissa, highlighted in the plots.
Interestingly, while some seeds still perform reasonably well despite the enormously high Q -value, it clearly correlates to the high variance in the cumulative reward plot.
C li
pp i n g &
A b s .
T r e a t m
[Figure]
n
[Figure]
C li
pp i n g &
N o
A b s .
T r e a t m e n t
N o
C li
pp i n g &
N o
A b s .
T r e a t m e n t
[Figure]
B. A BLATION S TUDY : I NFLUENCE OF FIXED T ARGETS AND E NTROPY C LIPPING
To show the effect of the fixed target (c.d., Section 3.5) and the entropy clipping (c.f., 3.7), we conducted a range of ablation studies for different versions of LSIQ on all Mujoco task. The results are shown in Figure 7 for the LSIQ version only with an entropy critic and in Figure 8 for the LSIQ version with an entropy and a regularization critic.
As can be seen from the Figures, the version with the fixed target and the entropy clipping performs at best. It is especially noteworthy that the entropy clipping becomes of particular importance on tasks that require a high temperature parameter β , which is the case for the Humanoid-v3 environment. [Figure] [Figure]
B. A LL E XPERIMENT R ESULTS OF THE DIFFERENT V ERSION OF LSIQ
Figure 9 presents all the plots presented in Figure 3 with the additional Humanoid-v3 results.
Figure 10 and Figure 11 correspond to Figure 4 but show all five environments.
The corresponding learning curves are shown in Appendix B.6 and B.5.
[Figure]
[Figure]
[Figure]
B. I MITATION L EARNING FROM S TATES O NLY – F ULL T RAINING C URVES
The learning curves for the learning from observation experiments can be found in Figure 12, 13,and 15 for 1 , 5 , 10 and 25 expert demonstrations, respectively
[Figure]
[Figure] [Figure]
[Figure] [Figure]
B. I MITATION L EARNING FROM S TATES AND A CTIONS – F ULL T RAINING C URVES
The learning curves for the experiments where states and actions are observed can be found in
Figure 16, 17, 18 and 19 for 1 , 5 , 10 and 25 expert demonstrations, respectively
[Figure]
HalfCheetah
[Figure] [Figure]
[Figure] [Figure]