R EINFORCEMENT L EARNING WITH S PARSE R EWARDS USING G UIDANCE FROM O FFLINE D EMONSTRATION
A BSTRACT
A major challenge in real-world reinforcement learning (RL) is the sparsity of reward feedback.
Often, what is available is an intuitive but sparse reward function that only indicates whether the task is completed partially or fully.
However, the lack of carefully designed, ﬁne grain feedback implies that most existing RL algorithms fail to learn an acceptable policy in a reasonable time frame.
This is because of the large number of exploration actions that the policy has to perform before it gets any useful feedback that it can learn from.
In this work, we address this challenging problem by developing an algorithm that exploits the ofﬂine demonstration data generated by a sub-optimal behavior policy for faster and efﬁcient online RL in such sparse reward settings.
The proposed algorithm, which we call the Learning Online with Guidance Ofﬂine (LOGO) algorithm, merges a policy improvement step with an additional policy guidance step by using the ofﬂine demonstration data.
The key idea is that by obtaining guidance from - not imitating - the ofﬂine data, LOGO orients its policy in the manner of the sub-optimal policy, while yet being able to learn beyond and approach optimality.
We provide a theoretical analysis of our algorithm, and provide a lower bound on the performance improvement in each learning episode.
We also extend our algorithm to the even more challenging incomplete observation setting, where the demonstration data contains only a censored version of the true state observation.
We demonstrate the superior performance of our algorithm over state-of-the-art approaches on a number of benchmark environments with sparse rewards and censored state.
Further, we demonstrate the value of our approach via implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance, where it shows excellent performance.
I NTRODUCTION
Reinforcement Learning (RL) is being considered for application to a variety of real-world settings that have very large state-action spaces, but even under the availability of accurate simulators determining how best to explore the environment is a major challenge.
Oftentimes, the problem setting is such that intuitively obvious reward functions are quite sparse.
Such rewards typically correspond to the attainment of a particular task, such as a robot attaining designated way points, with little ﬁne-grain reward feedback on the intermediate steps taken to complete these tasks.
The sparsity of rewards implies that existing RL approaches often do not make much headway into learning viable policies unless provided with carefully hand-tuned reward feedback by a domain expert.
Many of these problems correspond to systems where data has been gathered over time using an empirically determined (sub-optimal) behavior policy, which contains information important to help bootstrap learning.
An additional caveat is that this behavior data might only contain measurements of a subset of the true state.
Is it possible to design an algorithm that has principled policy optimization for efﬁcient learning, while also being able to explore by utilizing the data derived from a behavior policy?
A natural candidate to begin with is the framework of policy gradient algorithms (Schulman et al., 2015;
Lillicrap et al., 2016; Haarnoja et al., 2018), which performs really well in the dense reward setting.
Can such a policy gradient algorithm be aligned with a behavior policy for guided exploration that is critical in the sparse reward setting?
In this work, we propose a principled approach for using the well known trust region policy optimization (TRPO) algorithm (Schulman et al., 2015) with ofﬂine demonstration data for guidance in a sparse reward setting.
Our choice of the TRPO approach is motivated by its analytical tractability and superior performance.
Our key insight is that we can utilize the trust region approach while being guided by the behavior data by virtue of two steps.
The ﬁrst step is identical to traditional TRPO to generate a candidate policy.
In the second step, the objective is to ﬁnd a policy closest to the behavior policy, subject to it being in the trust region of the candidate generated in the ﬁrst step.
Thus, the second step ensures that the policy chosen is always guided by the behavior policy, but the level of alignment with the behavior policy can be reduced by shrinking the trust region over time.
We call our approach Learning Online with Guidance Ofﬂine (LOGO) to capture this two-step procedure.
This principled approach enables us to provide analytical performance guarantees while achieving superior performance in a number of benchmark problems.
Our main results are as follows: (i) LOGO can guarantee a performance improvement as long as the behavior policy is able to offer an advantage.
Reducing the alignment with the behavior policy over time allows us to extract this advantage and then smoothly move beyond to ﬁnd near-optimal policies, (ii) we provide a generalized version of the Performance Difference Lemma (Kakade &
Langford, 2002) wherein the stage reward function can depend on the policy itself, and use this result to determine a surrogate objective function for step 2 of LOGO that is straightforward to evaluate.
This allows us to implement LOGO in the manner of two TRPO-like steps, enabling us to leverage the TRPO code base, (iii) we show on standard MuJoCo environments that LOGO trained with sparse rewards can attain nearly the same performance as an optimal algorithm trained with dense rewards.
We go further and show that this excellent performance carries over to waypoint tracking by a robot over Gazebo simulations and real-world TurtleBot experiments, (iv) ﬁnally, we show that LOGO can also be used in the case where the demonstration data only contains a censored version of the true state (incomplete state information) by simply adding a projection to the available subset of the state space in step 2 of the algorithm.
Again, we provide supporting evidence via excellence in both
MuJoCo simulations, as well as obstacle avoidance by a TurtleBot that is trained with a behavior policy without a Lidar (censored state), but is tested with it (full state).
1. R ELATED W ORK
Our work is mainly related to two RL research areas:
Imitation Learning (IL): The goal of an IL algorithm is to imitate an (expert) policy using the demonstration data generated by that policy.
Behavior cloning (BC) is a simple IL approach where the expert policy is estimated from the demonstration data using supervised learning.
BC algorithms, however, suffer from the problem of distribution shift (Ross et al., 2011).
Inverse reinforcement learning (IRL) algorithms (Ng & Russell, 2000; Ziebart et al., 2008) estimate a reward function from the demonstration data and solve a forward RL problem using this reward function.
Generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) avoids the reward estimation problem by formulating the IL problem as a distribution matching problem, and provides an implicit reward for the RL algorithm using a discriminator (Goodfellow et al., 2014).
Most IL algorithms do not use reward feedback from the environment, and hence are restricted to the performance of the policy that generated the demonstration data.
Our approach is different from pure IL, and we leverage online RL with (sparse) reward feedback for efﬁcient learning.
Learning from Demonstration (LfD):
The key idea of the LfD algorithms is to use demonstration data to aid online learning (Schaal et al., 1997).
Many works propose to exploit demonstration data by adding it to the replay buffer with a prioritized replay mechanism to accelerate learning (Hester et al., 2018; Vecerik et al., 2017; Nair et al., 2018).
Rajeswaran et al.
(2018) combine a policy gradient algorithm with demonstration data by using a mix of behavior cloning and online RL ﬁne tuning.
Nair et al.
propose AWAC algorithm to accelerate online RL by leveraging large amounts of ofﬂine data with associated rewards.
Different from this, LOGO does not need the reward observations.
Moreover, we give provable guarantees on the performance of LOGO whereas AWAC does not have any such provable guarantee (further details are provided in Appendix G).
Kim et al.
(2013) propose a framework to integrate LfD and approximate policy iteration by formulating a coupled constraint convex optimization problem, where the expert demonstrations deﬁne a set of linear constraints.
This approach is, however, limited to small problems with a discrete action space.
The closest to our work is the PofD algorithm proposed by Kang et al.
PofD modiﬁes the reward function by taking a weighted combination of the sparse reward from the online interaction and an implicit reward obtained from the demonstration data using a discriminator.
Very different from this, we propose an intuitive and principled approach of using the ofﬂine demonstration data for guiding the online exploration during the initial phase of learning.
Our two step approach enables us to provide a rigorous performance guarantee for the proposed LOGO algorithm, and to leverage trust region-based approaches to solve high dimensional problems with even sparser settings.
We provide an extensive comparison between our algorithm and PofD in Section 5.
Ofﬂine RL: Recently, there has been many interesting works in the area of ofﬂine RL (Kumar et al., 2019; Fujimoto et al., 2019; Siegel et al., 2020; Wu et al., 2019a) which focus on learning a policy using only the ofﬂine data without any online learning or online ﬁne-tuning.
Different from these,
LOGO is an online RL algorithm (further details are provided in Appendix G).
P RELIMINARIES AND P ROBLEM S ETTING 2. P RELIMINARIES
A Markov Decision Process (MDP) is denoted as a tuple < S , A , R, P, γ > , where S is the state space,
A is the action space, R : S ×
A → R is the reward function, P : S ×
A × S → [0 , 1] is the transition probability function with P ( s (cid:48) | s, a ) giving the probability of transitioning to state s (cid:48) when action a is taken at state s , and γ ∈ (0 , 1) is the discount factor.
A policy π is a mapping from S to probability distribution over A , with π ( s, a ) specifying the probability of taking action a in state s .
A policy π can generate state-action trajectory τ , where τ = ( s 0 , a 0 , s 1 , a 1 , . . . )
, s 0 ∼ µ, a t ∼ π ( s t , · ) , s t +1 ∼ P ( ·| s t , a t ) and µ is the initial state distribution.
The inﬁnite horizon discounted return of policy π is deﬁned as J R
[Equation] [Equation]
0 γ t R ( s t , a t )] .
The goal of a reinforcement learning algorithm is to learn the optimal policy π (cid:63) =
arg max π J R ( π ) .
The value function of a policy π deﬁned as V πR ( s ) =
E τ ∼ π
[ (cid:80) ∞ t =
γ t R ( s t , a t ) | s 0 =
s ] , is the expected cumulative discounted reward obtained by following the policy π starting from the state s .
The action-value function of a policy π is deﬁned similarly as Q πR ( s, a ) =
E τ ∼ π
[ (cid:80) ∞ t =
γ
[Equation]
( s t , a
[Equation]
) | s 0 = s,
[Equation]
0 = a ] .
The advantage function is deﬁned as A πR ( s, a ) =
Q πR ( s, a ) − V πR ( s ) .
The discounted state visitation distribution for the policy π , denoted as d π , is deﬁned as d π ( s ) =
(1 − γ ) (cid:80) ∞
t =0 γ t P ( s t = s | π ) , where the probability is deﬁned with respect to the randomness induced by π, P and µ .
Deﬁnitions and Notations: The Kullback-Leibler (KL) divergence between two distribution p and q is deﬁned as D KL ( p, q ) =
(cid:80) x p ( x ) log p ( x ) q ( x ) .
The average KL divergence between two polices π 1 and π 2 with respect to d π 1 is deﬁned as D π 1 KL ( π 1 , π 2 ) =
E s ∼ d π
[ D KL ( π 1 ( s, · ) , π 2 ( s, · ))] .
The maximum KL divergence between two polices π 1 and π 2 is deﬁned as D maxKL ( π 1 , π 2 ) =
max s D KL ( π 1 ( s, · ) , π 2 ( s, · )) .
The total variation (TV) distance between two distribution p and q is deﬁned as D TV ( p, q ) =
(1 / 2) (cid:80) x | p ( x ) − q ( x ) | .
The average TV distance and the maximum TV distance between two polices π 1 and π 2 are deﬁned as D π 1 TV ( π 1 , π 2 ) =
E s ∼ d π
[ D TV ( π 1 ( s, · ) , π 2 ( s, · ))] and D max TV ( π 1 , π 2 ) = max s D TV ( π 1 ( s, · ) , π 2 ( s, · )) , respectively.
2. P ROBLEM S ETTING
As described in the introduction, our goal is to develop an algorithm that can exploit ofﬂine demonstration data generated using a sub-optimal behavior policy for faster and efﬁcient online reinforcement learning in a sparse reward setting.
Formally, we assume that the algorithm has access to the demonstration data generated by a sub-optimal behavior policy π b .
We ﬁrst consider the setting in which the demonstration data has complete state observation.
In this setting, the demonstration data has the form D
= { τ i } ni =1 , where τ i =
( s i 1 , a i 1 , . . .
, s iT , a iT ) , τ i ∼ π b .
Later, we also propose an extension to the incomplete observation setting in which only a censored version of the true state is available in demonstration data.
More precisely, instead of the complete state observation s i t , the demonstration data will only contain ˜ s it
=
o
( s it ) , where o ( · ) is a projection to a lower dimensional subspace.
We represent the incomplete demonstration data as ˜ D
= { ˜ τ i } ni =1 , where ˜ τ i =
(˜ s i 1 , a i 1 , . . . ,
˜ s iT , a iT ) .
We make the following assumption about the behavior policy π b .
Assumption 1.
In the initial episodes of learning, E a ∼ π b
[ A π k R ( s, a )] ≥ β > 0 , ∀ s, where π k is the learning policy employed by the algorithm in the k th episode of learning.
Intuitively, the above assumption implies that taking action according to π b will provide a higher advantage than taking actions according to π k because E a ∼
π k
[ A π k R ( s, a )] = 0 .
This is a reasonable assumption, since the behavior policies currently in use in many systems are likely to perform much better than an untrained policy.
We also note that a similar assumption is made by Kang et al.
(2018) to formalize the notion of a useful behavior policy.
We emphasize that π b need not be the optimal policy, and π b could be such that J R ( π b ) <
J R ( π (cid:63) ) .
Our proposed algorithm learns a policy that performs better than π b through online learning with guided exploration.
A LGORITHM AND P ERFORMANCE G UARANTEE 3. A LGORITHM
In this section, we describe our proposed LOGO algorithm.
Each iteration of our algorithm consists of two steps, namely a policy improvement step and a policy guidance step.
In the following, π k denotes the policy after k iterations.
Step 1: Policy Improvement:
In this step, the LOGO algorithm performs a one step policy improvement using the Trust Region Policy Optimization (TRPO) approach (Schulman et al., 2015).
This can be expressed as
[Equation]
The TRPO update ﬁnds the policy π k + 1 / 2 that maximizes the objective while constraining this maximizing policy to be within the trust region around π k deﬁned as { π : D π k KL ( π, π k ) ≤ δ } .
The TRPO approach provides a provable guarantee on the performance improvement, as stated in
Proposition 1.
We omit the details as this is a standard approach in the literature.
Step 2: Policy Guidance: While the TRPO approach provides an efﬁcient online learning strategy in the dense reward setting, it fails when the reward structure is sparse.
In particular, it fails to achieve any signiﬁcant performance improvement for a very large number of episodes in the initial phase of learning due to the lack of useful reward feedback that is necessary for efﬁcient exploration.
We propose to overcome this challenge by providing policy guidance using ofﬂine demonstration data after each step of the TRPO update.
The policy guidance step is given as
[Equation]
Intuitively, the policy guidance step aids learning by selecting the policy π k +1 in the direction of the behavior policy π b .
This is achieved by ﬁnding a policy that minimizes the KL divergence w.r.t.
π b , but at the same time lies inside the trust region around π k + 1 / 2 deﬁned as { π : D maxKL ( π, π k + 1 / 2 ) ≤ δ k } .
This trust region-based policy guidance is the key idea that distinguishes LOGO from other approaches.
In particular, this approach gives LOGO two unique advantages over other state-of-the-art algorithms.
First, unlike imitation learning that tries to mimic the behavior policy by directly minimizing the distance (typically KL/JS divergence) between it and the current policy, LOGO only uses the behavior policy to guide initial exploration.
This is achieved by starting the guidance step with a large value of the trust region δ k , and gradually decaying it according to an adaptive update rule (speciﬁed in
Appendix F) as learning progresses.
This novel approach of trust region based policy improvement and policy guidance enables LOGO to both learn faster in the initial phase by exploiting demonstration data, and to converge to a better policy than the sub-optimal behavior policy.
Second, from an implementation perspective, the trust region based approach allows to approximate the objective by a surrogate function that is amenable to sample based learning (see Proposition 2).
This allows us to implement LOGO in the manner of two TRPO-like steps, enabling us to leverage the TRPO code base.
Details are given Section 4.
3. P ERFORMANCE G UARANTEE
We derive a lower bound on the performance improvement, J R ( π k +1 ) − J R ( π k ) for LOGO in each learning episode.
We analyze the policy improvement step and policy guidance step separately.
We begin with the performance improvement due to the TRPO update (policy improvement step).
The following result and its analysis are standard in the literature and we omit the details and the proof.
Proposition 1 (Proposition 1, (Achiam et al., 2017)) .
Let π k and π k + 1 / 2 are related by (1) .
Then,
[Equation]
where (cid:15) R,k = max s,a | A π k R ( s, a ) | .
We now give the following result which can be used to obtain a lower bound on the performance improvement due to the policy guidance step.
Lemma 1.
Let π k + 1 / 2 be a policy that satisﬁes Assumption 1.
Then, for any policy π ,
[Equation]
Where (cid:15) R,k + 1 / 2 = max s,a | A π k +1 / 2 R (
[Equation]
|
[Equation]
Remark 1 .
Lemma 1 also gives an intuitive explanation for our proposed policy guidance step.
The policies used in the initial phase of the learning can be far from the optimal.
So, it is reasonable to assume that π k (and hence π k + 1 / 2 ) satisﬁes Assumption 1 in the initial phase of learning.
Then, minimizing D π KL ( π, π b ) can get a non-negative lower bound in (4), which will imply that the performance of π k +1 is better than π k + 1 / 2 .
This is indeed the idea behind the policy guidance step.
Combining the results of Proposition 1 and Lemma 1, and with some more analysis, we get the following performance improvement guarantee for the LOGO algorithm.
Theorem 1.
Let π k and π k + 1 / 2 are related by (1) and let π k + 1 / 2 and π k +1 are related by 2.
Let (cid:15) R,k and (cid:15) R,k + 1 / 2 be as deﬁned in Proposition 1 and Lemma 1, respectively.
Let R max = max s,a | R ( s, a ) | .
( i )
If
π
k +
/ 2 satisﬁes Assumption 1, then
[Equation]
( ii )
If π k + 1 / 2 does not satisfy Assumption 1, then
[Equation]
Remark 2 .
In the initial phase of learning when the baseline policy is better than the current policy, the policy guidance step can add a non-negative term to the standard lower bound obtained by the
TRPO approach, as shown in (5).
This indicates faster learning in the initial phase as compared to the naive TRPO approach.
The policy improvement guarantee during the later phase of learning is given by (6), which shows that LOGO achieves similar performance guarantee as TRPO when δ k
= O ( √ δ ) .
We ensure this by decreasing the value of δ k as the learning progresses.
Thus, Theorem 1 clearly shows the key advantage of the LOGO algorithm achieved by the novel combination of the policy improvement step and the policy guidance step.
P RACTICAL A LGORITHM
We ﬁrst develop an approximation to the policy guidance step (2) that is amenable to sample-based learning and can scale to policies paramaterized by neural networks.
This step involves minimizing
D π KL ( π, π b ) under a trust region constraint.
However, this is not easy to solve directly by a samplebased learning approach, because estimating it requires samples generated according to any possible π , which is clearly infeasible.
To overcome this issue, inspired by the surrogate function idea used in the TRPO algoirthm (Schulman et al., 2015), we derive a surrogate function for D π KL ( π, π b ) that can be estimated using only the samples from the policy π k + 1 / 2 .
For deriving a surrogate function for D π KL ( π, π b ) that is amenable to sample-based learning, we ﬁrst deﬁne the policy dependent reward function C π as C π
( s, a )
= log
( π ( s, a ) /π
b ( s, a )) .
Using C π , we can also deﬁne the quantities J C π (˜ π ) , V ˜ πC π , Q ˜ πC π and A ˜ πC π for any policy ˜ π , exactly as deﬁned in
Section 2.1 by replacing R by C π .
Using these notations, we now present an interesting result which we call the performance difference lemma for policy dependent reward function.
Lemma 2.
For any policies π and ˜ π ,
[Equation]
Note that the above result has an additional term, (1 − γ ) − 1 D π KL ( π, ˜ π ) , compared to the standard performance difference lemma (Kakade & Langford, 2002).
In addition to being useful for analyzing our algorithm, we believe that the above result may also be of independent interest.
We now make an interesting observation that J C π ( π ) =
(1 − γ ) − 1 D π KL ( π, π b ) (proof is given in the
Appendix), which can be used with Lemma 2 to derive the surrogate function given below.
Proposition 2.
Let π k + 1 / 2 be as given in (1) .
Then, for any policy π that lies in the trust region around π k + 1 / 2 deﬁned as { π : D maxKL ( π, π k + 1 / 2 ) ≤ δ k } , we have
[Equation]
where α k = D π k +1 / 2 KL
[Equation]
Now, instead of minimizing D π KL ( π, π b ) , we will minimize the upper bound as a surrogate objective function.
Note that only one term in this surrogate objective function depends on π , and that term can be estimated using only samples from π k + 1 / 2 .
Thus, we will approximate the policy guidance step as
[Equation]
Since D maxKL is difﬁcult to implement in practice, we will further approximate it by average KL divergence D π k +1 / 2 KL ( π, π k + 1 / 2 ) .
We note that this is a standard approach used in the TRPO algorithm.
We can now put steps (1) and (9) together to obtain the full algorithm.
However, as the policies are represented by large neural networks, solving them exactly is challenging.
Hence, for sufﬁciently small δ, δ k , we can further approximate the objective functions and constraints by a Taylor series expansion to obtain readily implementable update equations.
This is a standard approach (Schulman et al., 2015; Achiam et al., 2017), and we only present the ﬁnal result below.
Consider the class of policies { π θ : θ ∈ Θ } where θ is the parameter of the policy.
Let θ k be the parameter corresponding to policy π k .
Then, the Taylor series expansion-based approximate solution of (1) and (9) yields the ﬁnal form of LOGO as follows:
[Equation]
where g k =
[Equation]
While it is straightforward to compute A πC π for any policy when the form of the baseline policy π b is known, it is more challenging when only the demonstration data D generated according to π b is available.
We overcome this challenge by training a discriminator using the demonstration data D and the data D k + 1 / 2 generated by the policy π k + 1 / 2 (Goodfellow et al., 2014; Ho & Ermon, 2016; Kang et al., 2018) that will approximate the policy dependent reward function C π k +1 / 2 .
Further details on training this discriminator and a concise form of the algorithm are given in Appendix D.
4. E XTENSION TO I NCOMPLETE O BSERVATION S ETTING
We now discuss how to extend LOGO to the setting where the behavior policy data contains only incomplete state observations.
For instance, consider the problem of learning a policy for a mobile robot to reach a target point without colliding with any obstacles.
Collision avoidance requires sensing the presence of obstacles, which is typically achieved by camera/Lidar sensors.
Learning an
RL policy for this problem requires a high ﬁdelity simulator that models various sensors and their interaction with the dynamics.
Such high ﬁdelity simulators are, however, typically slow and difﬁcult to parallelize, and training an RL policy using such a simulator in a sparse reward environment can be very time consuming and computationally expensive.
Often, it is much easier to train an RL policy for the trajectory tracking problem using only a simple kinematics simulator model which only has a lower dimensional state space compared to the original problem.
In particular, such a kinematics simulator will only have the position and velocity of the robot as the state instead of the true state with high dimensional camera/Lidar image.
Can we use the demonstration data or behavior policy from this low dimensional simulator to accelerate the RL training in a high dimensional/ﬁdelity simulator in sparse reward environments?
We answer this question afﬁrmatively using a simple idea to extend
LOGO into such incomplete observation settings.
Since the behavior policy appears in LOGO only through a policy dependent reward function
C π ( s, a ) = log( π ( s, a ) /π
b ( s, a )) , we propose to replace this with a form that can handle the incomplete observation setting.
For any state s ∈ S , let ˜ s
= o ( s ) be its projection to a lower dimensional state space ˜ S .
As explained in the mobile robot example above, a behavior policy ˜ π b in the incomplete observation setting can be interpreted as mapping from ˜ S to the set of probability distributions over A .
We can then replace C π ( s, a ) in the LOGO algorithm with ˜ C
π ( s, a ) deﬁned as ˜ C π ( s, a ) =
log( π ( s, a ) / ˜
π b ( o ( s ) , a )) .
When only the demonstration data with incomplete observation is available instead of the representation of the behavior policy, we propose to train a discriminator to estimate ˜ C π .
Let ˜ D
= { ˜ τ i } ni =1 be the demonstration with incomplete observation, where ˜ τ i
=
(˜ s i 1 , a i 1 , . . . ,
˜ s iT , a iT ) .
Then we train a discriminator using ˜ D and D π to estimate ˜ C π .
E XPERIMENTS
We now evaluate LOGO from two perspectives: (i) Can LOGO learn near-optimally in a sparse reward environment when guided by demonstration data generated by a sub-optimal policy?
(ii) Can LOGO retain near-optimal performance when guided by sub-optimal and incomplete demonstration data with sparse rewards?
We perform an exhaustive performance analysis of LOGO, ﬁrst through simulations under four standard (sparsiﬁed) environments on the widely used MuJoCo platform (Todorov et al., 2012).
Next, we conduct simulations on the Gazebo simulator (Koenig & Howard, 2004) using
LOGO for way-point tracking by a robot in environments with and without obstacles, with the only reward being attainment of way points.
Finally
[Figure]
we transfer the trained models to a real-world
TurtleBot robot (Amsters & Slaets, 2019) to demonstrate LOGO in a realistic setting
[Figure]
MuJoCo full ofﬂine observation.
[Figure]
v
[Figure]
r a g
[Figure]
e p
i s o d
i c r e w a r d
[Figure]
R e w a r d
InvertedDoublePendulum-v
[Figure] [Figure]
In what follows, we present a summary of our experiments 1 and results.
We remark that LOGO is relatively easy to implement and train, since its two TRPO-like steps imply that we can utilize much of a TRPO code base.
5. M U J O C O S IMULATIONS
In our ﬁrst set of experiments, we consider four standard environments using the MuJoCo platform.
We introduce sparsity by reducing the events at which reward feedback is provided.
Speciﬁcally, for
Hopper, HalfCheetah and Walker2d, we provide a reward of +1 at each time only after the agent moves forward over 2 , 20 , and 2 units from its initial position, respectively.
For InvertedDoublePendulum, we introduce sparsity by providing a reward only at the end of the episode.
Besides LOGO, our candidate algorithms are as follows: (i) Expert: We train TRPO in the dense reward environment to provide the optimal baseline, (ii) Behavior: We use a partially trained expert that is still at a sub-optimal stage of learning to provide behavior data, (iv) GAIL: We use Generative
Adversarial Imitation Learning (Ho & Ermon, 2016), which attempts to imitate the Behavior policy (iii) TRPO:
We directly use TRPO in the sparse reward setting without any guidance from behavior data (iv) POfD: We use Policy Optimization from Demonstration (Kang et al., 2018) as a heuristic approach to exploiting behavior data.
(v) BC-TRPO:
We warm start TRPO by performing behavior cloning (BC) on the sub-optimal behavior data.
(vi) DAPG:
We use Demo Augmented Policy
Gradient (Rajeswaran et al., 2018) which warm starts Natural Policy Gradient (NPG) algorithm using
BC, and ﬁne tunes it online using behavior data in a heuristic manner.
Note that for all algorithms, we evaluate the ﬁnal performance in the corresponding dense reward environment provided by OpenAI
Gym, which provides a standardized way of comparing their relative merits.
Setting 1: Sparse Rewards.
We compare the performance of our candidate algorithms in the sparse reward setting in Figure 1a, which illustrates their rewards during training.
As expected, TRPO fails to make much meaningful progress during training, while GAIL can at best attain the same performance as the sub-optimal behavior policy.
While BC-TRPO beneﬁts from warm starting, it fails to learn later due to the absence of online guidance as in the case of LOGO.
POfD and LOGO both use the behavior data to boot strap learning.
POfD suffers from the fact that it is inﬂuenced throughout the learning process by the behavior data, which prevents it from learning the best policy.
However, LOGO’s nuanced exploration using the demonstration data only as guidance enables it to quickly attain optimality.
LOGO outperforms DAPG in all but one environment.
Setting 2: Sparse Rewards and Incomplete State.
We next consider sparse rewards along with reducing the revealed state dimensions in the behavior data of Setting 1.
These state dimensions are selected by eliminating state dimensions revealed until the expert TRPO with dense rewards starts seeing reduced performance.
This ensures that we are pruning valuable information.
Since GAIL,
POfD and LOGO all utilize the behavior data, we use the approach of projecting the appropriate reward functions that depend on the behavior data into a lower dimensional state space described in
Section 4.1.
We emphasize that BC-TRPO and DAPG cannot be extended to this setting as they require full state information for warm starting (in BC-TRPO and DAPG) and online ﬁne tuning (in
DAPG).
We see from Figure 1b that LOGO is still capable of attaining good performance, although training duration is increased.
We will further explore the value of being able to utilize behavior data with such incomplete state information in robot experiments in the next subsection.
5. T URTLE B OT E XPERIMENTS
We now evaluate the performance of LOGO in a real-world using TurtleBot, a two wheeled differential drive robot (Amsters & Slaets, 2019).
We train policies for two tasks, (i) Waypoint tracking and (ii)
Obstacle avoidance, on Gazebo, a high ﬁdelity 3D robotics simulator.
Task 1: Waypoint Tracking: The goal is to train a policy that takes the robot to an arbitrary waypoint within 1 meter of its current position in an episode of 20 seconds.
The episode concludes when the robot either reaches the waypont or the episode timer expires.
The state space of the agent are its relative x, y, coordinates and orientation φ to the waypoint.
The actions are its linear and angular velocities.
The agent receives a sparse reward of +1 if it reaches the waypoint, andotherwise.
We created a sub-optimal Behavior policy by training TRPO with dense rewards on our own low ﬁdelity kinematic model Python-based simulator with the same state and action space.
While it shows reasonable waypoint tracking, the trajectories that it generates in Gazebo are inefﬁcient, and its real-world waypoint tracking is poor (Figures 2 (d)-(f)).
As expected, TRPO shows poor performance in this sparse reward setting and often does not attain the desired waypoint before the episode concludes, giving the impression of aimless circling seen in Figure 2(f).
LOGO is able to effectively utilize the Behavior policy and shows excellent waypoint tracking seen in Figures 2 (d)-(f).
[Figure]
visual reference in
Task 2: Obstacle Avoidance: The goal and rewards are the same as in Task 1, with a penalty of −for collision with the obstacle, shown in Figure 2 (a)-(c).
The complete state space is now augmented by a 2D Lidar scan in addition to coordinates and orientation described in Task 1.
However, the Behavior policy is still generated via the low ﬁdelity kinematic simulator without the obstacle, i.e., it is created on a lower dimensional state space.
As seen in Figures 2 (g)-(i), this renders the Behavior policy impractical for Task 2, since it almost always hits the obstacle in both Gazebo and the realworld.
However, it does possess information on how to track a waypoint, and when combined with the full state information, this nugget is utilized very effectively by LOGO to learn a viable policy as seen in Figures 2 (g)-(i).
Further, TRPO in this sparse reward setting does poorly and often collides with the obstacle in real-world experiments as seen in Figure 2 (i).
C ONCLUSION
In this paper, we studied the problem of designing RL algorithms for problems in which only sparse reward feedback is provided, but ofﬂine data collected from a sub-optimal behavior policy, possibly with incomplete state information is also available.
Our key insight was that by dividing the training problem into two steps of (i) policy improvement and (ii) policy guidance using the ofﬂine data, each using the concept of trust region based policy optimization, we can both obtain principled policy improvement and desired alignment with the behavior policy.
We designed an algorithm entitled
LOGO around this insight and showed how it can be instantiated in two TRPO-like steps.
We both proved analytically that LOGO can exploit the advantage of the behavior policy, as well as validated its performance through both extensive simulations and illustrative real-world experiments.
A CKNOWLEDGEMENT
This work was supported in part by the National Science Foundation (NSF) grants NSF-CRII-CPS1850206, NSF-CAREER-EPCN-2045783, NSF-CPS-2038963 and NSF-CNS and U.S.
Army Research Ofﬁce (ARO) grant W911NF-19-1-0367.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of the sponsoring agencies.
A U SEFUL T ECHNICAL R ESULTS
We will use the well known Performance Difference Lemma in our analysis (Kakade & Langford, 2002)
Lemma 3 (Performance Difference Lemma) .
For any two policies π and ˜ π ,
[Equation]
The following result gives a bound on the difference between discounted state visitation distributions of two policies in terms of their average total variation difference.
Lemma 4 (Lemma 3, (Achiam et al., 2017)) .
For any two policies π and π (cid:48) ,
[Equation]
We will use the following well known result.
Lemma 5.
For any two policies π and π
[Equation]
Proof.
From Pinsker’s inequality, D TV ( p, q ) ≤ (cid:112) D KL ( p, q ) / 2 for any two distributions p, q .
Combining this with Jensen’s inequality, we get D π TV ( π, π (cid:48) )
≤ (cid:112) D π KL ( π, π (cid:48) ) / 2 .
We will use the following result which is widely used in literature.
We provide the proof for completeness.
Lemma 6.
Given any policy π and a function f π : S ×
A → R ,
[Equation]
Proof.
[Equation]
π [ ∞ (cid:88) t
= γ t f π ( s t , a t )]
= (cid:88) s,a ∞ (cid:88) t
= γ t P ( s t =
s, a t = a | π ) f π ( s, a ) =
(cid:88) s,a ∞ (cid:88) t
= γ t P ( s t = s | π ) π ( s, a )
f π ( s, a ) =
(cid:88) s,a π ( s, a )
f π ( s, a ) ∞ (cid:88) t
= γ t P ( s t = s | π )
= (1 − γ ) − 1 (cid:88) s,a d π ( s ) π
( s, a )
f π ( s, a ) =
(1 − γ ) − 1 E s ∼ d π ,a ∼ π ( s, · ) [ f π ( s, a )] .
B P ROOF OF THE R ESULTS IN S ECTIONB. P ROOF L EMMA
Proof.
Starting from the performance difference lemma,
J R ( π ) − J R ( π k + 1 / 2 ) =
(1 − γ ) − 1 E s ∼ d π ,a ∼ π ( s, · )
[ A π k +1 / 2 R ( s, a )]
= (1 − γ ) − 1 (cid:88) s
d π ( s ) (cid:88) a π b ( s, a ) A π k +1 / 2 R ( s, a ) + (1 − γ ) − 1 (cid:88) s d π ( s ) (cid:88) a ( π ( s, a ) − π b ( s, a ))
A π k +1 / 2 R ( s, a ) ( a ) ≥ (1 − γ ) − 1 β + (1 − γ ) − 1 (cid:88)
s
d π ( s ) (cid:88) a ( π ( s, a ) − π b ( s, a ))
A π k +1 / 2 R ( s, a )
[Equation]
where ( a ) follows from the premise that π k + 1 / 2 satisﬁes
Assumption 1, ( b ) is obtained by denoting (cid:15) R,k + 1 / 2 = max s,a | A π k +1 / 2 R ( s, a ) | , and ( c ) follows from Lemma 5.
B. P ROOF T HEOREM
We will ﬁrst prove the following result.
Lemma 7.
For any two policies π and ˜ π ,
[Equation]
where R max = max s,a | R ( s, a ) | .
Proof.
We have,
[Equation]
where ( a ) follows from Lemma 4.
The lower bound can also be obtained in a similar way.
Combining both, we obtain the desired result.
Proof of Theorem 1.
( i )
If π k + 1 / 2 satisﬁes Assumption 1, then (5) follows immediately by combining the results of Proposition 1 and Lemma 1.
( ii )
If π k + 1 / 2 does not satisfy Assumption 1, we can directly bound J R ( π k +1 )
− J R ( π k + 1 / 2 ) without invoking Lemma 1 (and hence the implicit assumption that moving towards π b improves the policy).
Use Lemma 7, we get
[Equation]
where the last inequality follows from the fact D maxTV ( π k +1 , π k + 1 / 2 ) ≤ δ k according to the policy guidance step (2).
Now, combining the result of Proposition 1 and the above inequality, we obtain (6).
C P ROOF OF THE R ESULTS IN S ECTIONC. P ROOF OF L EMMA
Proof.
We follow the proof technique of the performance difference lemma (PDL) with some modiﬁcations to get the result.
For any given initial state s ,
[Equation] [Equation]
where ( a ) is obtained by rearranging the summation and telescoping.
The ﬁrst summation in (12) can now be handled as in the proof of the PDL
[Equation]
and we omit the details.
So
[Equation]
we get
[Equation]
The second summation can be written as
[Equation]
where ( b ) is obtained by using Lemma 6.
Now, by taking expectation on the both sides of (12) w.r.t.
the initial state distribution µ , and then substituting (13) and (14) there, we get the desired result.
C. P ROOF OF P ROPOSITION
Lemma 2 provides an approach to evaluate the inﬁnite horizon return J C π ( π ) of policy π using the inﬁnite horizon return J C ˜ π
(˜ π ) and advantage A ˜ πC ˜ π of policy ˜ π .
Unfortunately, the RHS of (7) contains two terms that requires taking expectation w.r.t.
d π , which means that empirical estimation requires samples according to π (that are unavailable).
This is a well known issue in the policy gradient literature and it is addressed by considering an approximation by replacing the expectation w.r.t.
d π by d ˜ π
(Kakade & Langford, 2002; Schulman et al., 2015).
We follow the same approach with minor modiﬁcations in the context of our problem.
We will ﬁrst prove the following lemmas.
Lemma 8.
For any two policies π and ˜ π ,
[Equation]
where (cid:15) ˜
π = max s,a | A ˜ πC ˜ π ( s, a ) |
[Equation]
Proof.
We will separately bound the two terms on the RHS of (7) in Lemma
[Equation]
Firstly,
[Equation] [Equation]
where ( a ) follows from Lemma 4 and ( b ) follows from Lemma 5.
Secondly, D π KL ( π, ˜ π ) ≤ D maxKL ( π, ˜ π ) .
Using this fact and the inequality (15) in (7), we get the desired result.
We now make an interesting observation that J C π ( π ) is a scaled version of the average KL divergence between π and π b .
We formally state this result below.
Lemma 9.
For any arbitrary policy π ,
[Equation]
Proof.
We have,
[Equation]
where ( a ) follows from Lemma 6.
Proof of Proposition 2.
Using the result of Lemma 9 in the inequality given by Lemma 8, for any policy π and ˜ π , we get
[Equation]
Now, replacing ˜ π by π k + 1 / 2 , we get
[Equation]
D
[Equation]
π, π
[Equation] [Equation]
where α k = D π k +1 / 2 KL
[Equation] [Equation]
Now, for any π that lies in the trust region { π : D maxKL ( π, π k + 1 / 2 ) ≤ δ k } , we have
D π k +1 / 2 KL ( π, π k + 1 / 2 ) ≤ D maxKL ( π, π k + 1 / 2 ) ≤ δ k .
Using this in (17) gives the ﬁnal result.
D D ETAILS OF THE P RACTICAL A LGORITHM
As explained in Section 4, when π b is known, obtaining C π ( s, a ) = log( π ( s, a ) /π
b ( s, a )) and estimating A πC π is straightforward.
Hence, LOGO can perform the policy improvement and policy guidance step according to the update equations given in (10) by directly using π b .
When the form of π b is unknown and we only have access to the demonstration data D generated according to π b , we have to estimate C π
( s, a )
using D and D π , where D π is the trajectory data generated according to π .
Instead of estimating log( π ( s, a ) /π
b ( s, a )) directly, we make use of the one-to-one correspondence between a policy and its discounted state-action visitation distribution deﬁned as ρ π ( s, a ) =
d π ( s ) π ( s, a ) (Syed et al., 2008).
More precisely, we estimate log( ρ π ( s, a ) /ρ
π b ( s, a )) using D and
D π instead of estimating log( π ( s, a ) /π
b ( s, a )) directly.
We can then use the powerful framework of generative adversarial networks (GAN) (Goodfellow et al., 2014) to estimate this quantity.
This can be achieved by training a discriminator function B : S ×
A → [0 , 1] as
[Equation]
We note that the GAN-based approach to estimate a distance metric between ρ π and ρ π b is popular in the imitation learning literature (Ho & Ermon, 2016; Kang et al., 2018).
The optimal discriminator for the above problem is given by B ∗ ( s, a ) =
[Equation]
(Goodfellow et al., 2014, Proposition 1).
Hence, given the discriminator function B obtained after training to solve (18), we use C π ( s, a )
= − log B ( s, a ) in the LOGO algorithm, which provides an approximation to the quantity of interest.
As shown in Section 5, this approximation yields excellent results in practice.
We summarize the LOGO algorithm below.
Algorithm 1 LOGO Algorithm
[Equation]
[Equation] [Equation]
[Equation] [Equation]
g k F − 1 k g k 12:
Decay δ k (according to the adaptive rule described in Appendix F) 13:
Estimate h k and L k using C π k and D π k
+
Perform policy guidance step: θ k +
=
θ k
+ 12 − (cid:115)
δ k h
Tk L −
k
h
k
L − 1 k h k 15: end for
E D ETAILS OF T URTLE B OT E XPERIMENTS
We evaluate the performance of LOGO in a real-world setting using TurtleBot, a two wheeled differential drive robot (Amsters & Slaets, 2019).
We consider two different simulators for training our policy.
The ﬁrst one is a simple kinematics-based low ﬁdelity simulator.
The second one is a sophisticated physics-based Gazebo simulator (Koenig & Howard, 2004) with an accurate model of the TurtleBot.
We use the low ﬁdelity simulator to get a sub-optimal Behavior policy by training TRPO with dense rewards.
The details are given in Section E.1.
Gazebo simulator is used for implementing the LOGO algorithm in a sparse reward setting with complete and incomplete observations.
The details are given in Section E.2.
E. L OW F IDELITY K INEMATICS S IMULATOR
We design the low ﬁdelity simulator using the OpenAI gym framework where the dynamics are governed by the following equations
[Equation]
where x t , y t are the coordinates of the bot, v t is the linear velocity, ω t is the angular velocity, and θ t is its yaw calculated from the quaternion angles, all calculated at time t .
∆ is the time discretization factor.
We deﬁne the state s t to be the normalized relative position w.r.t.
the waypoint, i.e., s t
=
(( x t − x w ) /G, ( y t − y w ) /G, θ t − θ wt ) , where x w , y w are the target coordinates of the waypoint, and θ wt is the target heading to the waypoint at time t .
we deﬁne the action a t to be the linear and angular velocities, i.e., a t =
[ v t , ω t ] .
We discretize the action space into 15 actions.
We handcraft a dense reward function as follows,
[Equation]
where d t is a combination of cross track and along track error deﬁned as follows,
[Equation]
[Equation]
H IGH F IDELITY G AZEBO S IMULATOR
Gazebo is a physics-based simulator with rendering capabilities and can be used for realistic simulation of the environment.
It is conﬁgurable and can be used to model robots with multiple joint constraints, actuator physics, gravity and frictional forces and a wide range of sensors in indoor as well as outdoor settings.
Gazebo facilitates close-to-real-world data collection from the robots and sensor models.
Since it runs in real-time, it takes millions of simulation frames for an RL agent to learn simple tasks.
Gazebo can also speed up simulation by increasing step size.
This can however lead to loss of precision.
For this project, we ran the training scheme on Gazebo in almost real time with a simulation step-size of ∆ T = 0 .
The Gazebo simulation setup consists of the differential drive robot (TurtleBot3 Burger) model spawned in either an empty space or a custom space with obstacles at a predeﬁned location.
A default coordinate grid is setup with respect to the base-link of the robot model.
A ROS (Stanford
Artiﬁcial Intelligence Laboratory et al.)
framework is instantiated using a custom-built OpenAI Gym environment which acts as an interface between the proposed RL algorithm and the simulation model.
ROS topics are used to capture the state update information of the robot asynchronously in a callback driven mechanism.
For the purpose of our experiments we use the following ROS topics,
1. /odom
(for x t , y t , θ t ): Odometry information based
on the wheel encoder
/cmd vel (for v t , ω t ): Linear and angular velocity commands 3.
/scan (for obstacle avoidance): Scan values from the Lidar
The /odom and /scan topics are used to determine s t while /cmd vel is used to output the required action a t .
The state space and action space for waypoint tracking task is similar to state and action space in section E.1.
For obstacle avoidance tasks, we also include the distance values obtained form the Lidar as a part of the state.
E. R OBOT P LATFORM : T URTLE B OT
We use TurtleBot 3 (Amsters & Slaets, 2019), an open source differential drive robot equipped with a RP Lidar as our robotic platform for real-world experiments.
We use ROS as the middleware to setup the communication framework.
The bot transmits its state (position and Lidar information) information over a network to a intel NUC, which transmits back the corresponding action according to the policy being executed.
E. T RAINING : W AYPOINT T RACKING
In navigation problems, we have a global planner that uses a high level map of the bot’s surroundings for planning a trajectory using waypoints a unit-meter distance from each other, while the goal of the bot is to achieve these waypoints.
To obtain a waypoint tracking scheme, we ﬁrst train a behavior policy on the low ﬁdelity simulator using TRPO to reach a waypoint approximately one unit distance (1m in the real world) away from its initial position.
In each training episode, the bot is reset to the origin and a waypoint is randomly chosen as x w ∼ uniform ([ − 1 , 1]) , y w = 1 .
The episode is terminated if the robot reaches the waypoint, or if it crosses the training boundary or exceeds the maximum episode length.
The behavior policy obtained after training in the low ﬁdelity simulator is then used in the LOGO algorithm training on Gazebo.
LOGO is trained in Gazebo with sparse sparse rewards, where a reward of +1 is provided if the bot reaches the waypoint, and 0 otherwise.
We evaluate the trained policy in Gazebo and the real world, by providing it a series of waypoints to track in order to reach its ﬁnal goal.
E. T RAINING : O BSTACLE A VOIDANCE
We train our bot for obstacle avoidance in Gazebo using the behavior policy described in the section above.
Our goal is to use the skills of waypoint navigation from the behavior policy to guide and learn the skills of obstacle avoidance.
The state space includes the Lidar scan values in addition to the state space described in section E.1.
The /scan provides 360 values, each of these indicate the distance to the nearest object in a 1 ◦ sector.
For the purpose of our experiments, we use the minimum distance in each 60 ◦ sector.
This reduces the Lidar data to 6 values.
We train our algorithm on Gazebo with a ﬁxed obstacle for random waypoints.
In each training episode, the bot is reset to the origin and a waypoint is generated similar to the previous section.
The episode is terminated if same conditions in the previous section are satisﬁed or if a collision with the obstacle occurs.
We demonstrate the performance of our algorithms both in Gazebo as well as the real-world.
F I MPLEMENTATION D ETAILS
We implement all the algorithms in this paper using PyTorch (Paszke et al., 2019).
For all our experiments, we have a two layered (128 × 128) fully connected neural network with tanh activation functions to parameterize our policy and value functions.
We use a learning rate of 3 × 10 − 4 , a discount factor γ = 0 . 99 , and TRPO parameter δ = 0 .
We decay the inﬂuence of the behavior policy by decaying δ k .
We start with δ 0 , and we do not decay δ k for the ﬁrst K δ iterations.
For k > K δ , we geometrically decay δ k as δ k ← αδ k , whenever the average return in the current iteration is greater than the average return in the past 10 iterations.
The rest of the hyperparameters for MuJoCo simulations, Gazebo simulation, and real-world experiments are given in table 1.
In table 2 we provide details on the demonstration data collected using the behavior policy.
We implement LOGO based on a publicly available TRPO code base.
We implement PofD, TRPO,
BC-TRPO, and GAIL by modifying the same code base as used by LOGO.
We run DAPG using code base and hyperparameters provided by the authors.
For consistency, we run all algorithms using the same batch size.
[Table] [Table]
G R ELATED W ORK
Ofﬂine RL: Recently, there have been many interesting works in the area of ofﬂine RL which use ofﬂine data to learn a policy.
In particular, ofﬂine RL algorithms such as BEAR (Kumar et al., 2019),
BCQ (Fujimoto et al., 2019), ABM (Siegel et al., 2020), and BRAC (Wu et al., 2019a) focus on learning a policy using only the ofﬂine data without any online learning or online ﬁne-tuning.
The key idea of these ofﬂine RL algorithms is to learn a policy that is ‘close’ to the behavior policy that generated the data via imposing some constraints, to overcome the problem of distribution shift.
This approach, however, often results in conservative policies, and hence it is difﬁcult to guarantee signiﬁcant performance improvement over the behavior policy.
Moreover, standard ofﬂine
RL algorithms are not immediately amenable to online ﬁne-tuning, as observed in Nair et al.
LOGO is different from the ofﬂine RL algorithms in the following two key aspects.
First, unlike the ofﬂine RL algorithms mentioned before, LOGO is an online RL algorithm which uses the ofﬂine demonstration data for guiding the online exploration during the initial phase of learning.
By virtue of this clever online guidance, LOGO is able to converge to a policy that is signiﬁcantly superior than the sub-optimal behavior policy that generated the demonstration data.
Second, ofﬂine RL algorithms typically require large amount of state-action data with the associated rewards .
LOGO only requires a small amount demonstration data since it is used only for the guiding the online exploration.
Moreover, LOGO requires only the state-action observation and does not need the associated reward data.
In many real-world application applications, it may be possible to get state-action demonstration data from human demonstration or using a baseline policy.
However, it is difﬁcult to assign reward values to these observation.
Advantage Weighted Actor-Critic (AWAC) algorithm
(Nair et al., 2020) propose to accelerate online
RL by leveraging ofﬂine data.
This work is different from our approach in four crucial aspects.
( i )
AWAC requires ofﬂine data with associated rewards whereas LOGO requires only the state-action observations (not the reward data).
In many real-world applications, it may be possible to get stateaction demonstration data from human demonstration or using a baseline policy.
However, it may be difﬁcult to assign reward values to these observations, especially in the sparse reward setting.
( ii )
AWAC explicitly mentions that it leverages large amounts of ofﬂine data whereas LOGO relies on small amount of sub-optimal demonstration data.
( iii ) LOGO gives a novel and theoretically sound approach using the double trust region structure that provides provable guarantees on its performance.
AWAC algorithm does not give any such provable guarantees.
( iv ) LOGO can be easily extended to the setting with incomplete state information where as AWAC is not immediately amenable to such extension.
There are also many recent works on addressing multiple aspects of IL and LfD, including combining
IL and RL for long horizon tasks (Gupta et al., 2019), learning from imperfect (Gao et al., 2018;
Jing et al., 2020; Wu et al., 2019b) and incomplete (Libardi et al., 2021; G¨ulc¸ehre et al., 2020) demonstrations.
Their approaches, performance guarantees and experimental settings are different from that of our problem and proposed solution approach.
[Figure] [Figure]
Samples 1e
A v e r a g e e p i s o d i c r e w a r d
Hopper-v
[Figure]
[Figure] [Figure]
α
[Figure]
1e
[Figure]
Hopper
[Figure]
v
[Figure]
H S ENSITIVITY A NALYSIS OF α AND K δ
We run LOGO for different values of α and K δ by keeping the other parameters ﬁxed on the Hopperv2 environment.
We observe from Figure 3a that the performance of LOGO is not sensitive to perturbations in α .
When α = 1 , we do not decay the inﬂuence of the behavior data, this results in a performance close to the behavior data which is inline with our intuition.
We observe from Figure 3b that LOGO is not sensitive to perturbations in K δ as well.
We observe that the performance of LOGO is similar for K δ = 1 , 50 , 100 .
This because we only decay δ k whenever the average return in the current iteration is greater than the average return in the past 10 iterations.
We further note that K δ controls the iteration from which we begin decaying the inﬂuence of the behavior data, this can be clearly seen for K δ = 500 , 1000 where the performance rises as soon as the decaying begins.