P OLICY G RADIENTS I NCORPORATING THE F UTURE
A BSTRACT
Reasoning about the future – understanding how decisions in the present time affect outcomes in the future – is one of the central challenges for reinforcement learning (RL), especially in highly-stochastic or partially observable environments
While predicting the future directly is hard in this work we introduce a method that allows an agent to “look into the future” without explicitly predicting it
Namely , we propose to allow an agent, during its training on past experience, to observe what actually happened in the future at that time, while enforcing an information bottleneck to avoid the agent overly relying on this privileged information
Coupled with recent advances in variational inference and a latent-variable autoregressive model, this gives our agent the ability to utilize rich and useful information about the future trajectory dynamics in addition to the present.
Our method, Policy
Gradients Incorporating the Future (PGIF), is easy to implement and versatile, being applicable to virtually any policy gradient algorithm.
We apply our proposed method to a number of off-the-shelf RL algorithms and show that PGIF is able to achieve higher reward faster in a variety of online and ofﬂine RL domains, as well as sparse-reward and partially observable environments
I NTRODUCTION
Fundamentally, reinforcement learning (RL) is composed of gathering useful information ( exploration ) and assigning credit to that information ( credit assignment ).
Both of these problems present their own unique learning challenges.
In this work, we focus on credit assignment, which refers to the challenge of matching observed outcomes in the future to decisions made in the past.
Humans appear to do this in a sample efﬁcient manner (Johnson-Laird, 2010), and so it is natural to expect our own RL agents to do so as well.
One of the most popular approaches to credit assignment, known as model-free RL, is to learn a value function to approximate the future return given a starting state and action.
The value function is learned using experience of the agent acting in the environment via temporal difference (TD) methods (Sutton, 1988), which regress the value function to a target based on a combination of groundtruth returns achieved in the environment and the approximate value function itself.
The need to bootstrap learning of the value function on its own estimates is known to lead to difﬁculties in practice, where one must achieve a careful balance between bias and variance (Harutyunyan et al., 2019; Weaver & Tao, 2001; Schulman et al., 2016; Mnih et al.).
If a slight imbalance arises, the consequences can be disastrous for learning (Tsitsiklis & Van Roy, 1996; van Hasselt et al., 2018; Sutton & Barto, 2018).
For example, in ofﬂine RL this issue is so pronounced that algorithms must apply strong regularizations on both learned policy and value function to achieve stable performance (Wu et al., 2020; Kumar et al., 2019; Zhang et al., 2021; Nachum et al., 2019).
The model-free approach plays dual to the model-based approach, where an agent learns a dynamics and reward model of the environment, and then learns an agent to optimize behavior in this model.
Thus, credit assignment boils down to utilizing an appropriate planning algorithm that can perform multiple rollouts in the model, effectively allowing the agent to “look into the future” (Racanière et al., 2017; Pascanu et al., 2017) to determine cause-and-effect (Sutton, 1991; Peng et al., 2018; Abbas et al.).
While model-based RL may appear more straightforward, learning an accurate model is a challenge in practice, presenting its own sample-efﬁciency problems (Wang et al., 2020) as well as memory and computational issues (Łukasz Kaiser et al., 2020).
Model-based approaches are thus most beneﬁcial when the environment exhibits some level of regularity (Fra, 2019).
Beyond these issues, credit assignment in both model-free and model-based RL is further exacerbated by partially observable environments, in which the full environment state is not known to the learning agent.
Thus it is infeasible to predict future events accurately.
When applied to such non-Markovian domains, model-free algorithms relying on bootstrapping and value function approximation tend to be biased (Singh, 1994).
On the other hand for model-based approaches, learning an accurate dynamics model in such domains is a difﬁcult, potentially ill-deﬁned problem (Suematsu & Hayashi, 1999; Bush & Pineau, 2009).
In this work, we aim to circumvent these challenges.
We propose a simple modiﬁcation to model-free
RL that allows the learned policy and value function to “look into the future” but without the need to learn an accurate model.
Namely, we propose to modify the policy and value function to not only condition on the presently observed state and action but also on the subsequent trajectory (sampled by the agent as it was interacting with the environment) following this state and action.
This way, our method mitigates potential approximation or feasibility issues in accurately modeling the future.
To ensure that the learned policy and value function remains relevant during inference (i.e., data collection) when the future trajectory is unavailable, we place an information bottleneck (Saxe et al., 2018; Tishby et al., 1999) on the additional inputs, encouraging the learned functions to minimize their reliance on this privileged information.
One may thus view our method as an instance of teacher forcing or Z -forcing (Goyal et al., 2017; Lamb et al., 2016) where our student is the learned policy and value function and the teacher is some function of the information in the future trajectory.
It is well known that combining a strong autoregressive decoder with latent variables while ensuring that they carry useful information is difﬁcult (Bowman et al., 2016; Chen et al., 2017).
Z-forcing enforces the incorporation of relevant information by learning latent variables that predict the future steps.
Practically, our method, Policy Gradients Incorporating the Future (PGIF), is versatile and easy to implement.
We use either a backwards RNN or a transformer to inject downstream information from the observed trajectories by way of latent variables, with a KL divergence regularization on these latents.
We apply PGIF on top of a variety of off-the-shelf RL algorithms, including RNNbased PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), and BRAC (Wu et al., 2020), and evaluate these algorithms on online and ofﬂine RL as well as sparse-reward and partially observable environments.
In all of these domains, we demonstrate the ability of PGIF to achieve higher returns faster compared to these existing RL algorithms on their own, thus showing that our proposed method is both versatile and beneﬁcial in practice.
B ACKGROUND AND N OTATION
We begin by providing a brief overview of the notation and preliminary concepts that we will use in our later derivations.
Markov Decision Processes (MDPs) MDPs are deﬁned by a tuple (cid:104)S , A , P , R, ρ 0 , γ (cid:105) where S is a set of states , A is a set of actions , P is a transition kernel giving a probability P ( s (cid:48) | s, a ) over next states given the current state and action, R : S ×
A → [ R min , R max ] is a reward function, ρ 0 is an initial state distribution, and γ ∈ [0 , 1) is a discount factor.
An agent in this MDP is a stationary policy π giving a probability π ( a | s ) over actions at any state s ∈ S .
A policy π interacts with the
MDP by starting at s 0 ∼ ρ
and then at time t ≥ 0 sampling an action a t ∼ π ( s t ) at which point the
MDP provides an immediate reward R ( s t , a t ) and transitions to a next state s t +1 ∼ P ( s t , a t ) .
The interaction ends when the agent encounters some terminal state s T .
The value function V π : S → R of a policy is deﬁned as V π ( s ) =
E π [ (cid:80) T − 1 t =
γ t r t | s
=
s ] , where E π denotes the expectation of following π in the MDP and T is a random variable denoting when a terminal state is reached.
Similarly, the state-action value function Q π : S ×
A → R is deﬁned as Q π ( s, a ) =
E π [ (cid:80) T − 1 t =
γ t r t | s
=
s, a 0 = a ] .
The advantage A π is then given by A π ( s, a ) =
Q π ( s, a ) − V π ( s ) .
We denote ρ π as the distribution over trajectories τ
= ( s 0 , a 0 , r 0 , . . .
, s T ) sampled by π when interacting with the MDP.
During learning, π is typically parameterized (e.g., by a neural network), and in this case, we use π θ to denote this parameterized policy with learning parameters given by θ .
The policy gradient theorem (Sutton et al., 1999) states that, in order to optimize the RL objective E s 0 ∼ ρ
[ V π ( s 0 )] , a parameterized policy should be updated with respect to the gradient of the following loss (ignoring any gradients through ρ π θ , where we denote stopping the gradient with ¯ θ ),
[Equation]
where ˆ Q t is an unbiased estimate of Q π ( s t , a t ) .
In the simplest case, ˆ Q t is the empirically observed future discounted return following s t , a t .
In other cases, an approximate Q -value or advantage function is incorporated to trade-off between the bias and variance in the policy gradients.
When the
Q or V value function is parameterized, we will use ψ to denote its parameters.
For example, the policy gradient loss with a parameterized Q ψ is given by,
[Equation]
The value function Q ψ is typically learned via some regression-based temporal differencing method.
For example,
[Equation]
Stochastic Latent Variable Models
In our derivations, we will utilize parameterized policies and value functions conditioned on auxiliary inputs given by stochastic latent variables.
That is, we consider a latent space Z , typically a real-valued vector space.
We deﬁne a parameterized policy that is conditioned on this latent variable as π θ ( a | s, z ) for a ∈ A , s ∈ S , z ∈ Z ; i.e., π θ takes in states and latent variables and produces a distribution over actions.
In this way, one can consider the latent variable z as modulating the behavior of π θ in the MDP.
During interactions with the
MDP or during training, the latent variables themselves are generated by some stochastic process, thus determining the behavior of π θ .
For example, in the simplest case z may be sampled from a latent prior p υ (Z) ( z | s ) , parameterized by υ (Z) .
Thus, during interactions with the MDP actions a t are sampled as a t ∼ π θ ( s t , z t ) , z t ∼ p
υ
(Z) ( s t ) .
We treat parameterized latent variable value functions analogously.
Speciﬁcally, we consider a latent space U and a parameterized Q -value function as Q ψ ( s, a, u ) .
A prior distribution over these latent variables may be denoted by p υ (U) ( u | s ) .
P OLICY G RADIENTS I NCORPORATING THE F UTURE
Our method aims to allow a policy during training to leverage future information for learning control.
We propose to utilize stochastic latent variable models to this end.
Namely, we propose to train π θ and Q ψ with latent variables ( z , u ) =
{ ( z t , u t ) } T − 1 t =0 sampled from a learned function q φ ( τ ) which has access to the full trajectory.
For example, the PGIF form of the policy gradient objective in (2) may be expressed as
[Equation]
The PGIF form of the temporal difference objective in (3) may be expressed analogously as,
[Equation]
It is clear that any RL objective which trains policies and/or value functions on trajectories can be adapted to a PGIF form in a straightforward manner.
For example, in our experiments we will apply
PGIF to an LSTM-based PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), and BRAC (Wu et al., 2020).
While the PGIF-style objectives above adequately achieve our aim of allowing a policy to leverage future trajectory information during training, they also present a challenge during inference.
When performing online interactions with the environment, one cannot evaluate q φ ( τ ) , since the full trajectory τ is not yet observed.
Therefore, while we want to give π θ and Q ψ the ability to look at the full τ during training, we do not want their predictions to overly rely on this privileged information.
To this end, we introduce a regularization on q φ in terms of a KL divergence from a prior distribution p υ
( τ ) :
= { p υ ( z t , u t | s t ) } T − 1 t =0 which conditions ( z t , u t ) only on s t .
Thus, in the case of policy gradient, the full loss is,
[Equation]
where β is the weight of the divergence term.
The introduction of this prior thus solves two problems: (1) it encourages the learned policies and value functions to not overly rely on information beyond the immediate state; (2) it provides a mechanism for inference, namely using latent samples from p υ ( s ) when interacting with the environment.
Parameterization of q φ
In our implementation, we parameterize q φ ( τ ) as an RNN operating in reverse order on τ .
Speciﬁcally, we use an LSTM network to process the states in τ backwards, to yield LSTM hidden states b = { b t } T − 1 t =0 .
The function q φ is then given by Gaussian distributions with mean and variance at time t derived from the backwards state b t .
In practice, to avoid potential interfering gradients from the objectives of π θ and Q ψ , we use separate RNNs with independent parameters q φ ( Z ) , q φ ( U ) for z t , u t , respectively.
See Figure 1 for a graphical diagram of the training and inference procedure for PGIF.
In our empirical studies shown in Appendix G, we will also show that a transformer (Vaswani et al., 2017) can be used in place of an RNN with minimal decrease in performance, providing more computational efﬁciency and potentially allowing for better propagation of information over time.
3. V ARIATIONAL I NFORMATION B OTTLENECK I NTERPRETATION
The KL regularization we employ above may be interpreted as a variational information bottleneck, constraining the mutual information between the latent variable distribution and the trajectory τ .
Here we provide a brief derivation establishing this equivalence.
For simplicity, we consider a speciﬁc timestep t ∈ N and a starting state s t =
s .
Let T ≥ t denote the random variable for all information contained after and including timestep t in trajectory τ induced by π .
Let U t be the random variable for latents u t induced by q φ ( τ ≥ t ) conditioned on all steps in the trajectory after and including t .
Consider a constrained objective minimizing J PGIF,TD while enforcing an upper bound I max on the mutual information between the distribution of trajectory steps and the distribution of latent variables
I ( T ≥ t , U t | s t = s ) .
This objective is given by,
[Equation]
Recall the deﬁnition of mutual information:
[Equation]
[Equation] where p ( u t | s t = s ) is the marginal distribution over the latent variable p ( u t | s t = s )
= (cid:82) q φ ( u t | τ ≥ t ) ρ π
( τ ≥ t | s t = s ) dτ ≥ t .
This marginal is intractable to compute directly, and so to approximate this marginal we introduce a variational distribution h ( u t | s t = s ) .
By deﬁnition we know that
D KL [ p ( u t | s t = s ) (cid:107)
h ( u t | s t = s )] ≥ 0 .
We can then see that (cid:82) p ( u t | s t = s ) log p ( u t | s t = s ) du t ≥ (cid:82) p ( u t | s t = s ) log h ( u t | s t = s ) du t .
We therefore derive the upper bound for use in equation 7 as,
[Equation]
We can subsume the constraint into the objective as,
[Equation]
By taking h to be our learned prior p υ , we recover the single step ( s t = s ), TD analogue of the PGIF objective in (6), offset by a constant β · I max , as desired.
3. Z-F ORCING WITH A UXILIARY L OSSES
While our proposed training architecture enables the policy and value function to look at the full trajectory τ , in practice it may be difﬁcult for the trajectory information to propagate, especially in settings with highly sparse learning signals.
In fact, it is known that such latent variable models may ignore the latent variables due to optimization issues, completely negating any potential beneﬁt (Bengio et al., 2015).
To circumvent these issues, we make use of the idea of Z-forcing (Goyal et al., 2021), which employs auxiliary losses and models to force the latent variables to encode information about the future.
We denote this loss as J Ax ( ζ ) where ζ is the set of parameters in any auxiliary models, and elaborate on the main forms of this loss which we consider below.
We do not combine different types of auxiliary losses in our total loss function.
We emphasize that these auxiliary losses are applied exclusively to the backwards encoder q φ ( τ ) and have no direct impact on the learned policy or value functions.
In fact, we conduct ablations in
Appendix F showing that these auxiliary losses applied directly to policy and value functions without PGIF perform worse.
State based forcing (Force)
A simple way to force state information to be encoded is to derive conditional generative models p ζ ( b t | z t ) over the backwards states given the inferred latent variables z t ∼ q φ ( Z ) ( z t | b t ) , and similarly for the latents u t .
We can write this auxiliary objective as a maximum log-likelihood loss J Ax ( ζ ) =
− E q φ ( Z ) ( z t | b t ) [log p ζ ( b t | z t )] .
This way, we enforce the noisy mapping b t → z t deﬁned by q φ (Z) to not be too noisy so as to completely remove any information from b t .
Value prediction networks (VPN) A more sophisticated approach to force information to be propagated is to use an autoencoder-like, model-based auxiliary loss.
To this end, we take inspiration from VPNs (Oh et al., 2017), and apply an auxiliary loss that uses b t to predict future rewards, values, and discounts.
Note that, in principle, b t already has access to this information, by virtue of the backwards RNN or transformer conditioned on the future trajectory.
Thus, this auxiliary loss only serves to enforce that the RNN or transformer dutifully propagates this information from its inputs.
We also note that, in contrast to the state based forcing described above, this approach only enforces b t to contain the relevant information, and it is up to the RL loss whether this information should be propagated to the latents z t , u t .
We give a detailed explanation of VPNs in Appendix B.
3. F ULL A LGORITHM
The full learning objective for PGIF is thus composed of three components: First, a latent-variable augmented RL objective, e.g., policy gradient as shown in (4).
Second, a KL regularizer, e.g., as shown in (6).
Finally, an auxiliary loss, given by either state based forcing (Force) or value prediction networks (VPN).
We present an example pseudocode of a PGIF-style policy gradient with learned value function in Algorithm 1.
See Appendix C for further details, including how to adaptively tune the coefﬁcients on the KL and auxiliary loss components as well as more speciﬁc pseudocode algorithms for advantage policy gradient and soft actor-critic.
Require: Initial parameters: θ, υ (U) , υ (Z) , φ (U) , φ (Z) , ψ, ζ PG , ζ TD , Weights: α PG , α TD , β PG , β TD 1: for policy-step k = 0 , 1 , 2 , . . .
, N do 2:
Collect set of Trajectories D
= { τ i , . . . } :
3: repeat 4: z t ∼ p
υ
( Z ) ( z t ) 5:
Execute: π θ ( a t | s t , z t ) and observe r t , s t +1 from environment.
6: until episode termination 7: for Trajectory: τ i ∈ D do 8: b Z = BackwardsLSTM Z ( τ i ) (Operates over the entire trajectory) 9: b U = BackwardsLSTM U ( τ i ) 10: τ i =
τ i ∪ { b Z , b U } 11: ∀{ s , a , r , b Z , b U } ∈ D : 12:
Derive J Ax-PG ( ζ PG ) , J Ax-TD ( ζ TD )
according to any auxiliary loss.
D [Equation] = [Equation]
( q
[Equation] ( u [Equation] [Equation]
= [Equation]
[Equation] [Equation] (cid:3) 15: [Equation]
|
[Equation] (cid:107)
[Equation]
[Equation] ( z
[Equation] s ))
J [Equation]
=
[Equation] (cid:3) 17:
Update all parameters w.r.t: J PG and J TD
R ELATED W ORK
We review relevant works in the literature in this section, with additional discussions in Appendix B.
Incorporating the future Recent works in model-based RL have considered incorporating the future by way of dynamically leveraging rollouts of various horizon lengths and then using them for policy improvement (Buckman et al., 2018) .
Z-forcing and stochastic dynamics models have been applied directly to learning environmental models and for behavioral cloning while incorporating the future but not for online or ofﬂine continuous control (Ke et al., 2019).
Our present work is unique for incorporating Z-forcing and conditioning on the future in the model-free RL setting.
A few other methods explore the future in less direct ways.
For example, RL Upside down (Schmidhuber, 2020) uses both reward (or desired return) and state to predict actions, turning RL into a supervised learning problem.
Hindsight Hindsight credit assignment introduces the notion of incorporating the future of a trajectory by assigning credit based on the likelihood of an action leading to an outcome in the future (Harutyunyan et al., 2019).
These methods were extended using a framework similar to ours, leveraging a backwards RNN to incorporate information in hindsight (Mesnard et al., 2021).
Still, there are a number of differences compared to our own work.
(1) Only the value function (rather than both the value and policy functions) is provided access to the future trajectory, whereas we show that allowing the actor access has beneﬁts in some tasks (Appendix F).
(2) There is no KL information bottleneck; rather information is constrained via an action prediction objective.
These previous works do not employ any Z-forcing, while it is well-known that learning useful latent variables in the presence of an autoregressive decoder is difﬁcult without Z-forcing (Bayer & Osendorfer, 2015); in fact, in our own preliminary experiments we found our algorithm performs signiﬁcantly worse without any auxiliary losses. Value driven hindsight modelling (HiMo) (Guez et al., 2020) proposes a hindsight value function, which is conditioned on future information in the trajectory. In this work, they primarily use the hindsight value function (separate from the agent’s value function) to learn a low-dimensional representation, which is distilled to a non-hindsight representation that is then used by the actual actor and critic. Thus there are a few key differences from our work: (1) There are no gradients passing from the RL loss to the representation loss (only the separate value prediction loss is used to train the representation), thus this method is arguably less end-to-end than PGIF; (2) The only mechanism for controlling the amount of information in the representation is through its dimension size and look-ahead, while using a KL penalty is more ﬂexible.
Our method is more versatile than previous works, being applicable to off-policy and ofﬂine RL settings rather than purely on-policy
RL, as in these previous works.
Nevertheless, it is an interesting avenue for future work to investigate how to combine the best of both approaches, especially with guarantees on variance reduction of policy gradient estimators (Nota et al., 2021) and hierarchical policies (Wulfmeier et al., 2021).
We now provide a wide array of empirical evaluations of our method, PGIF, encompassing tasks with delayed rewards, sparse rewards, online access to the environment, ofﬂine access to the environment, and partial observability.
In the appendix, we include further demonstrations of PGIF applied to the challenging AntMaze environment (Sec. H) with substantial performance improvements, online RL with full observability (Sec. E) with improvements over a SAC baseline, and numerous ablation analyses (Sec.
F, I) identifying the components of PGIF that are responsible for performance.
In our online RL experiments, we compare against Soft Actor Critic (SAC) (Haarnoja et al., 2018) and Proximal Policy Optimization (PPO) (Schulman et al., 2017) with an LSTM layer in the policy network.
The comparison with PPO is particularly important since this method also leverages a form of artiﬁcial memory of the past (but not forward-looking like in PGIF).
SAC is a state-of-the-art model-free RL method that employs a policy entropy term in a policy gradient objective and shows optimal performance and stability when compared with other online deep RL benchmarks.
We show the hyper-parameters for each experiment in the Appendix D.
In addition, we explore using our method with a transformer as opposed to an LSTM as the backwards network in Appendix G.
5. C REDIT A SSIGNMENT AND S PARSE R EWARDS
PGIF aims to provide a better representation for credit assignment by accounting for downstream information rather than current. This can be used as additional input to the value function or policy. We ﬁnd this helps address two very related problems, optimal credit assignment and learning with sparse rewards.
The former problem is exasperated by the latter since rewards are rarely obtained, possibly only at the end of the episode.
The relevance of each action ( credit ) must be assigned using only these rare rewards. [Table] The BSuite score is calculated in terms of the regret normalized [ random , optimal ] → [0 , 1] (higher is better).
The number after ± is the standard deviation.
We ﬁrst aim to show that our method is effective in a simple environment where credit assignment is the paramount objective.
We examine the UmbrellaLength task from BSUITE (Osband et al., 2020), a task involving a long sequential episode where only the ﬁrst observation (a forecast of rain or shine) and action (whether to take an umbrella) matter, while the rest of the sequence contains random unrelated information.
A reward of +1 is given at the end of the episode if the agent chooses correctly to take the umbrella or not depending on the forecast.
This difﬁcult task is used to test the agent’s ability to assign credit correctly to the ﬁrst decision.
We evaluate PGIF-versions of SAC and PPO to vanilla discreteSAC (Haarnoja et al., 2018; Christodoulou, 2019) as well as a PPO-LSTM (Schulman et al., 2017) baseline.
Results are presented in Table 1.
We see that our method is able to achieve best performance.
This suggests that the PGIF agent is able to efﬁciently and accurately propogate information about the ﬁnal reward to the initial timestep, more so than either the one-step backups used in SAC or the multi-step return regressions used in PPO-LSTM can.
We continue to the Gym-MiniGrid (Chevalier-Boisvert et al., 2018) set of partially-observable environments, for which rewards are sparse and there is a non-zero reward only when the agent completes the task (with the reward proportional to the time taken). The agent is only given a local ego-centric view of its environment.
For example, in DoorKey a key must ﬁrst be obtained and then a door opened leading to another room with the goal state.
It is difﬁcult, but essential, to assign credit to obtaining the key and opening the door in the correct sequence.
We compare against PPO-LSTM and an implementation of HiMo, which also suggests to use information from the future during agent learning (Guez et al., 2020).
We show the results of these experiments and examples of the grids in Figure 2, with PGIF-PPO outperforming both baselines. The difference is especially stark on the Unlock Environment where credit assignment is the key challenge and the standard PPO baseline fails to converge to any rewards.
In LavaGap, there is no signiﬁcant improvement. We hypothesize that this is because the most common trajectories generated during early training are ones where the agent enters the lava. There is also no negative reward for entering the lava, and the episode is simply terminated, making it more difﬁcult to avoid.
This creates a situation where our future representation is highly suboptimal, especially during early training.
5. P ARTIAL O BSERVABILITY
We now aim to show that our method is not only effective in fully-observed Markovian settings, but also in environments with partial observability.
This set of experiments uses the MuJoCo robotics simulator (Todorov et al., 2012) suite of continuous control tasks.
These are a set of popular environments used in both online and ofﬂine deep RL works (Fujimoto et al., 2019; 2018) and provides an easily comparable benchmark for evaluating algorithm sample efﬁciency and reward performance.
As in previous work (Yang & Nachum, 2021), we introduce an easy modiﬁcation to these tasks to make the environment partially observable thereby increasing the difﬁculty: We zero-out a random dimension of the state space at each data collection step.
This helps us test partial observability, a feature common in robotic agent tasks.
We compare a PGIF-style SAC implementation to vanilla SAC and PPO-LSTM on these domains.
We show the results of these experiments in Fig.
We ﬁnd that PGIF can provide improved performance on these difﬁcult tasks, suggesting that PGIF is able to leverage future information in the trajectory to appropriately avoid uncertainties about the environment, more so than when only conditioning on the immediate state (vanilla SAC) or even when conditioning on the entire past trajectory (PPO-LSTM).
Interestingly, we ﬁnd that the simple state based forcing (Force) performs more consistently better than the more sophisticated VPN based forcing.
See Appendix E for online evaluations without partial observability.
5. O FFLINE RL E VALUATIONS
To asses if our method is effective in an ofﬂine RL setting, we evaluate our proposed algorithm in several continuous control ofﬂine RL tasks (Fu et al., 2020) against Behavior Regularized Actor Critic (BRAC) (Wu et al., 2020) and Batch-Constrained Q-learning (BCQ) (Fujimoto et al., 2019).
BRAC operates as a straightforward modiﬁcation of SAC, penalizing the value function using a measure of divergence (KL) between the behaviour policy and the learned agent policy.
For our PGIF algorithm, we use BRAC as the starting point.
For these ofﬂine MuJoCo tasks, we examine D4RL datasets classiﬁed as medium (where the training of the agent is ended after achieving a "medium" level performance) and medium expert (where medium and expert data is mixed) (Fu et al., 2020).
Datasets that contain these sub-optimal trajectories present a realistic problem for ofﬂine RL algorithms.
We also include an ofﬂine version of the AntMaze, which is particularly challenging due to sparse rewards.
We show the results of these experiments in Table 2.
We ﬁnd that our method outperforms the baselines in all but one of the tasks in terms of ﬁnal episodic reward.
We hypothesize in the medium-expert setting that we perform slightly worse due to the lack of action diversity which makes learning a dynamics representation difﬁcult.
Interestingly, in contrast to the online results, we ﬁnd that VPN based forcing performs better than state based forcing, although even state based forcing usually performs better than the baseline methods. [Table] 2731 ±3250 ±2980 ±1851 ± [Table]
2483 ±3048 ±2431 ±2010 ±
[Table]
1757 ±2327 ±1930 ±1722 ±
[Table]
3687 ±3989 ±3821 ±2653 ± [Table] 5462 ±6037 ±6231 ±4722 ± [Table]
5580 ±5418 ±5491 ±4463 ± [Table]  D ISCUSSION In this work, we consider the problem of incorporating information from the entire trajectory in model-free online and ofﬂine RL algorithms, enabling an agent to use information about the future to accelerate and improve its learning.
Our empirical results attest to the versatility of our method.
The beneﬁts of our method are apparent in both online and ofﬂine settings, which is a rare phenomenon given that many previous ofﬂine RL works suggest that what works well in online RL often transfers poorly to ofﬂine settings, and vice versa (Fujimoto et al., 2019).
Beyond just online and ofﬂine RL, our results encompass partial observability, sparse rewards, delayed rewards, and sub-optimal datasets, demonstrating the ability for PGIF to achieve higher reward faster in all settings.
We also wish to highlight potential risks in our work.
Speciﬁcally, the use of future information in
PGIF may exacerbate biases present in the experience or ofﬂine dataset.
For example, it is known that expert datasets lack action diversity.
Further conditioning on the future in this dataset could force these biases to be incorporated more easily into the learning algorithm.
Some other biases that may arise are a trajectory that contains some marginal reward and is therefore incorporated into our policy/value function with PGIF.
This could hamper exploration of the agent and prevent discovery of states that yield optimal reward.
It may be interest to combine our method with exploration strategies.
Perhaps there are beneﬁts in decreasing access to future information during initial exploratory agent steps so the agent is better able to explore.
Furthermore, we ﬁnd that our method is slower to train than the baselines we compare to, due to the fact that the architecture requires training an LSTM, with at times, very long trajectories as input.
We must use the entire trajectories to train this LSTM. These issues can largely be alleviated by using a transformer, with minimal difference in performance.