E FFICIENT P LANNING IN A C OMPACT L ATENT A CTION S PACE
Meta AI (FAIR)
A BSTRACT
Planning-based reinforcement learning has shown strong performance in tasks in discrete and low-dimensional continuous action spaces.
However, planning usually brings significant computational overhead for decision making, so scaling such methods to high-dimensional action spaces remains challenging.
To advance efficient planning for high-dimensional continuous control, we propose Trajectory
Autoencoding Planner (TAP), which learns low-dimensional latent action codes from offline data.
The decoder of the VQ-VAE thus serves as a novel dynamics model that takes latent actions and current state as input and reconstructs longhorizon trajectories.
During inference time, given a starting state, TAP searches over discrete latent actions to find trajectories that have both high probability under the training distribution and high predicted cumulative reward.
Empirical evaluation in the offline RL setting demonstrates low decision latency which is indifferent to the growing raw action dimensionality.
For Adroit robotic hand manipulation tasks with high-dimensional continuous action space, TAP surpasses existing model-based methods by a large margin and also beats strong model-free actor-critic baselines.
I NTRODUCTION
Planning-based reinforcement learning (RL) methods have shown strong performance on board games (Silver et al., 2018; Schrittwieser et al., 2020), video games (Schrittwieser et al., 2020;
Ye et al., 2021) and low-dimensional continuous control (Janner et al., 2021).
Planning conventionally occurs in the raw action space of the Markov Decision Process (MDP), by rolling-out future trajectories estimated by a dynamics model of the environment, which is either predefined or learned.
While such a planning procedure is intuitive, planning in raw action space can be inefficient.
Firstly, planning in a high-dimensional raw action space can lead to suboptimal performance.
The optimal plan in a high-dimensional raw action space can be difficult to find, and most of the rollouts can be irrelevant.
Even if the optimizer is powerful enough to find the optimal plan, it is still difficult to make sure the learned model is accurate in the whole raw action space.
In such cases, the planner can exploit the weakness of the model and lead to over-optimistic planning.
Secondly, planning in raw action space means the planning procedure is tied to the temporal structure of the environment.
The conventional planner in raw action space predicts trajectories in the raw temporal domain, which can be slow and can lead to compounding prediction errors.
However, human planning is much more flexible, for example, humans can introduce temporal abstractions and plan with high-level actions; humans can plan backward from the goal; the plan can also start from a high-level outline and get refined step-by-step.
The limitations of raw action space planning cause slow decision speeds, which hamper adoption in real-time control.
[Figure]
In this paper, we propose the Trajectory Autoencoding Planner (TAP), which learns a latent action space and latent-action model from offline data.
A latent-action model takes state s 1 and latent actions z as input and predicts a segment of future trajectories τ
= ( a 1 , r 1 , R 1 , s 2 , a 2 , r 2 , R 2 , ... ) .
This latent action space can be much smaller than the raw action space since it only captures plausible trajectories in the dataset, preventing out-of-distribution actions.
Furthermore, the latent action decouples the planning from the original temporal structure of MDP.
This enables the model, for example, to predict multiple steps of future trajectories with a single latent action.
We evaluate TAP extensively in the offline RL setting.
Our results on low-dimensional locomotion control tasks show that TAP enjoys competitive performance as strong model-based, model-free actor-critic, and sequence modelling baselines.
On tasks with higher dimensionality, TAP not only surpasses model-based methods like MOPO (Yu et al., 2020) Trajectory Transformer(TT)
(Janner et al., 2021)
but also significantly outperforms strong model-free ones (e.g., CQL (Kumar et al.
2020)and IQL (Kostrikov et al., 2022)).
In Figure 1(c), we show how the relative performance between TAP and baselines changes according to the dimensionality of the action space.
One can see that the advantage of TAP starts to pronounce when the dimensionality of raw actions grows and the margin becomes large for high dimensionality, especially when compared to the model-based method TT.
This can be explained by the innate difficulty of policy optimization in a high-dimensional raw action space, which is avoided by TAP since its planning happens in a low-dimensional discrete latent space.
At the same time, the sampling and planning of TAP are significantly faster than prior work that also uses Transformer as a dynamics model: the decision time of TAP meets the requirement of deployment on a real robot (20Hz) (Reed et al., 2022), while
TT is much slower and the latency grows along the state-action dimensionality in Figure 1(b).
M ETHOD
To enable flexible planning in learned latent action space, we propose Trajectory Autoencoding
Planner (TAP) based on the VQ-VAE.
In this section, we will describe the latent action model framework and provide the model architecture details for TAP.
Then we elaborate on how to plan with this latent-action model.
2. L ATENT -A CTION M ODEL AND P LANNING
Consider the following trajectory τ of length T , sampled from an MDP with a fixed stochastic behaviour policy, consisting of a sequence of states, actions, rewards and return-to-go R t = (cid:80) i =
t γ i − t r i as proxies for future rewards:
[Equation]
We model the conditional distribution of the trajectory p ( τ | s , z ) with a sequence of latent variables z
= ( z 1 , . . .
, z M ) .
Assume the state and latent variables ( s , z ) can be deterministically mapped to a trajectory τ so that p ( τ | s , z )
= 1 ( τ = h ( s , z )) p ( z | s ) .
We refer to z as latent actions and p ( z | s ) as the latent policy .
The mapping h ( s , z ) from state and latent actions ( s , z ) to a trajectory τ is thus a latent-action model .
In a deterministic MDP, the trajectory for an arbitrary h ( s , z ) with p ( z | s ) > 0 will be an executable plan, namely, the trajectory can be recovered by following the action sequences, starting from state s .
Therefore, we can optimize latent actions z in order to find an optimal plan.
Planning in the latent action space has two advantages compared to planning in the raw action space.
Firstly, the latent-action model only captures possible actions in the support of the behaviour policy, allowing the latent action space to potentially be much smaller than the raw action space.
For example, considering the policy is a mixture of X policies, then the latent action space can be a discrete space with only X actions, no matter how high-dimensional the raw action space is.
In addition, only allowing the in-distribution actions prevents the planner from exploiting the weakness of the model by querying the actions with high uncertainty whose values are most susceptible to overestimation.
Secondly, the latent-action model allows the decoupling of the temporal structure between planning and modelling.
When planning in the raw action space, the time resolution of planning must be the same as the predicted trajectories.
In contrast, planning in the latent action space can be much more flexible, as a latent action does not have to be tied to a particular step of the transition.
This property allows the length of the latent action sequence M to be smaller than the planning horizon T , leading to more efficient planning.
2. L EARNING A L ATENT -A CTION M ODEL WITH VQ-VAE S
A discrete action space makes dealing with the full distribution of actions easier and also allows a spectrum of advanced planning algorithms to be applied Silver et al.
; Tillmann et al.
To learn a compact discrete latent action space, we propose Trajectory Autoencoding Planner (TAP) to model the trajectories with a state-conditioned VQ-VAE.
We treat x t :=
( s t , a t , r t , R t ) as a single token for the Transformer encoder and decoder.
Both the autoencoder and the prior over latent variables are conditioned on the first state s 1 of the trajectory.
Encoder g and decoder h .
For the encoder, token x t is processed by a causal Transformer, leading to T feature vectors.
We then apply a 1-dimensional max pooling with both kernel size and stride of L , followed by a linear layer.
This results in M = T/L vectors (¯ x ¯ x ¯ x 1 , ¯ x ¯ x ¯ x 2 , ..., ¯ x ¯ x ¯ x M ) , corresponding to the number of discrete latent variables.
After vector quantization (van den Oord et al., 2017), we get embedding vectors for the latent variables ( z z z 1 ,z z z 2 , ...,z z z M ) .
For the decoder, the latent variables are then tiled L times to match the number of the input/output tokens T .
For L = 3 , this is written as tile( zzz 1 ,zzz 2 , ...,zzz T/ 3 ) =
( zzz 1 ,zzz 1 ,zzz 1 ,zzz 2 ,zzz 2 ,zzz 2 , ...,zzz T/ 3 ,zzz T/ 3 ,zzz T/ 3 ) .
We concatenate the state and embedding vector for the codes and apply a linear projection to provide state information to the decoder.
After adding the positional embedding, the encoder then reconstructs the trajectory ˆ τ :=
( ˆ x 1 , ˆ x 2 , . .
, ˆ x T ) , where ˆ x t :=
(ˆ s t , ˆ a t , ˆ r t , ˆ R t ) .
Finally, to train the encoder/decoder, the reconstruction loss is the mean squared error over the input trajectories { τ } and reconstructed ones { ˆ τ } .
Prior distribution of latent codes .
While the derivation of VQ-VAE assumes a uniform prior over latents, in practice, learning a parameterised prior over latents will usually lead to better sample quality.
Similarly, TAP learns a prior policy p ( zzz | sss 1 ) to inform sampling and planning.
TAP uses a Transformer to autoregressively model the distribution of the latent action codes given the initial state sss 1 .
The state information is also blended into the architecture by the concatenation of token embeddings.
We denote the autoregressive prior policy as p ( zzz t | zzz <t ,sss 1 ) =
p ( zzz t | sss 1 ,zzz 1 ,zzz 2 , ...,zzz t − 1 ) .
2. P LANNING IN THE D ISCRETE L ATENT A CTION S PACE
We now describe how to plan with the learned TAP model.
This includes how to evaluate trajectories and optimize latent actions.
Evaluation Criterion .
Given the initial state and latent actions ( s 1 , z ) , we obtain a sample trajectory ˆ τ =
(ˆ x 1 , . .
, ˆ x T ) from decoder h , where ˆ x t :=
(ˆ s t , ˆ a t , ˆ r t , ˆ R t ) .
We score τ with the following function g :
[Equation]
The first part (in red) of the score function is the predicted return-to-go following the action sequences in the decoded trajectory ˆ τ .
The second part (in blue) is a term that penalizes out-ofdistribution plans, where the prior distribution p gives low probabiilty.
The hyperparameter α is set to be larger than the maximum of the discounted returns to select a plausible trajectory when the conditional probability of latent action sequences is lower than the threshold β M .
When the probability is higher than the threshold, the objective will then encourage choosing a high-return plan.
Beam Search in the Latent Space
We use causal Transformers for encoder, decoder and the prior policy, preventing the decoding depending on the future latent actions.
This allows us to partially decode the trajectory of length NL with first N latent variables.
Therefore we can apply a planning algorithm like beam search for more efficient optimization by only expanding promising branches of the search tree.
TAP beam search always keeps B best partial latent code sequences, and samples
E new latent actions conditioned on the partial codes.
Here B is the beam width and E is the expansion factor.
The pseudocode of the TAP beam search is shown in Algorithm 1 in the Appendix.
In practice, we find TAP with beam search is usually computationally more efficient because it can find better trajectories with fewer queries to the prior model and decoder.
E XPERIMENTS
The empirical evaluation of TAP consists of three sets of tasks from D4RL (Fu et al., 2020): gym locomotion control, AntMaze, and Adroit.
We compare TAP to a number of prior offline RL algorithms, including both model-free actor-critic methods (Kumar et al., 2020; Kostrikov et al., 2022) and model-based approaches (Kidambi et al., 2020;
Yu et al., 2020;
Lu et al., 2022).
Our work is conceptually the most related to the Trajectory Transformer (TT; Janner et al. 2021), a model-based planning method that predicts and plans in the raw state and action space, so this baseline serves as our main point of comparison.
Gym locomotion tasks serve as a proof of concept in the lowdimensional domain to test if TAP can accurately reconstruct trajectories for use in decision-making and control.
We then test TAP on Adroit, which has a high state and action dimensionality that causes TT to struggle because of its autoregressive modelling of long token sequences.
Finally, we also test TAP on AntMaze, a sparse-reward continuous-control problem.
TAP achieves similar performance as TT on AntMaze, surpassing model-free methods, where the details can be found in Appendix I.
3. R ESULTS
On low-dimensional gym locomotion control tasks, all the model-based methods show comparable performance to model-free ones.
However, the performance of model-based baselines on high- dimensional adroit tasks is much worse than the low-dimensional case, showing inferior scalability with respect to state and action dimensions.
In contrast, TAP shows consistently strong performance among low-dimensional and high-dimensional tasks and surpasses other model-based methods with a large margin on Adroit.
To show how the relative performance between TAP and baselines vary with action dimension, we average the relative performance on tasks of the same action dimensionality and plotted them in
Figure 1(c).
The x-axis is the action dimensionality of the tasks and the horizontal red line shows the relative performance of 1, meaning two methods achieve the same score.
We can see that TAP scales better in terms of decision latency compared to TT.
Also, TAP shows better performance for tasks with higher action dimensionality, compared to both TT and strong model-free actor-critic methods (CQL/IQL).
This can be explained by the increasing difficulty of policy optimization in larger action space due to the curse of dimensionality.
In contrast, TAP does the optimization in a compact latent space with a handful of discrete latent variables.
[Table]
3. D ECISION L ATENCY
Transformer-based trajectory generative models (such as TT and Gato (Reed et al., 2022)) treat each dimension of the state and action as an individual token.
Denoting S as the dimensionality of the state space and A as that of the action space
, TT requires D = S + A + 2 tokens to model a step of a trajectory.
Since the time complexity of the Transformer is O ( N 2 ) for a sequence of length N , the computational cost of TAP is significantly lower especially when the state-action dimensionality is high.
Specifically, for a trajectory with T steps, TAP uses T tokens (in autoencoders) and TT uses
TD tokens.
Therefore, the inference cost of TAP is O
( T 2 ) rather than O ( D 2 T 2 ) of TT, leading to more efficient training and planning.
Besides the computational complexity of the forward pass and sampling, we also tested how exactly the computational costs and decision latency of TT and TAP grow along with the state-action dimensionality of the tasks.
Tests are done on a platform with an i5 12900K CPU and a single RTX
GPU.
We fix the planning horizon to be 15 (the default planning horizon for TT) and test the decision latency on hopper ( D = 14 ), halfcheetah ( D = 23 ), ant ( D = 37 ) and Adroit-pen ( D = 71 ) tasks.
Even for the task with the lowest dimensionality, TT needs 1 .
5 seconds to make a decision and the latency grows to 32 seconds when dealing with adroit.
On the other hand, TAP manages to make a decision within 0 .
05 seconds
and meets the real-world robotics deployment criterion proposed by Reed et al.
Moreover, the latency remains nearly constant for different dimensionalities, making it computationally feasible to be applied to high-dimensional control tasks.
A R ELATED W ORK
Model-based RL TAP fits into a line of work on model-based RL method (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015; Wang & Ba, 2020; Schrittwieser et al., 2020; Amos et al., 2021; Hafner et al., 2021) because it makes decisions by predicting into the future.
In such approaches, prediction often occurs in the original state space of an MDP, meaning that models take a current state and action as input and return distribution over future states and rewards.
TAP is different from these conventional modelbased methods since the TAP decoder does not strictly follow the MDP causal structure but takes latent variables and the current state as inputs, returning a whole trajectory.
These latent variables act as a learned latent action space which is compact and allows efficient planning.
Planning in the Latent Space Hafner et al.
(2019) and Ozair et al.
(2021) proposed to learn a latent state space and dynamics function in the latent space.
However, in their cases, the action space of the planner is still the same as the raw MDP and the unrolling of the plan is still tied to the original temporal structure of the environment.
Wang et al.
; Yang et al.
(2021) proposed to learn representations of actions on-the-fly for black-box optimization and path planning setting.
While related at a high level, these works operate on environment dynamics that are assumed to be known in advance.
TAP extends this idea to the RL setting where the true dynamics of the environment are unknown, requiring a dynamics model and latent action representation space to be learned jointly.
RL as Sequence Modelling TAP follows a recent line of work that treats RL as a sequence modelling problem (Chen et al., 2021; Janner et al., 2021; Zheng et al., 2022).
Such works have used
GPT-2 (Radford et al., 2019) style Transformers (Vaswani et al., 2017) to model the whole trajectory of states, actions, rewards and values, and turn the prediction ability into a policy.
DT applies an
Upside Down RL (Schmidhuber, 2019) approach to predict actions conditioned on rewards.
TT applies planning in order to find the best trajectory that gives the highest return.
The strong sequence modelling power of the Transformer enables TT to generate long-horizon plans with high accuracy.
However, since planning is in the raw state and action space, the decision latency becomes a bottleneck in high-dimensional environments.
TAP provides a solution for TT and all other discrete space planning algorithms to efficiently plan in complex action spaces.
Offline RL In this paper, TAP is tested in an offline RL setting (Ernst et al., 2005) where we are not allowed to use online experience to improve our policy.
A major challenge of offline RL is to prevent out-of-distribution (OOD) actions to be chosen by the learned policy, so that the inaccuracy of the value function and the model will not be exploited.
Conservatism has been proposed as a solution to this issue (Kumar et al., 2020; Fujimoto & Gu, 2021; Lu et al., 2022; Kostrikov et al.,
2022; Kidambi et al., 2020).
TAP can naturally prevent the OOD actions (or, more generally, OOD sequences) because it models trajectories generated by behaviour policy.
To further discourage OOD trajectories from being selected as a plan, we add an OOD penalty given by the learned prior to the planning objective.
Hierarchical RL In the literature of hierarchical RL, a conceptually relevant work to ours is (CoReyes et al., 2018), which uses a VAE to project the state sequences into continuous latent variables and conditions policies on not only the current state but also the latent variable.
Whereas TAP models actions, states, rewards, and values jointly with a unified Transformer, Co-Reyes et al.trains a state decoder and a step-by-step policy decoder separately.
The policy decoder in their case has to be trained with a conventional RL algorithm with approximate gradients, whereas TAP follows the sequence modelling approach and is trained with fully (self-)supervised signals.
Since the high-level latent actions are discrete, our work is also related to RL methods in the options framework (Sutton et al., 1999; Stolle & Precup, 2002;
Bacon et al., 2017) since both the latent codes of TAP and options introduce a mechanism for temporal abstraction.
Conceptually, a latent action model can fully decouple the planning from the environment’s temporal structure and go beyond the feedforward shooting-based planning (see Appendix H for details).
In TAP, the latent actions emerge from the training of a generative model of trajectories, causing action segments to be grouped into the same discrete latent action if they lead to similar future trajectories.
Such a latent action space is a natural fit for model-based RL and efficient planning.
B A BALATION
TAP is a novel model-based RL method which involves several designs that can potentially be transferred to other model-based or offline RL methods.
In order to provide more insights into these design choices, we provide analyses here, together with ablations of common hyper-parameters like planning horizon.
In Figure 3, we showed a summary of the results of ablation studies on gym locomotion tasks.
The full results for more hyper-parameters and scores for each individual task can be found in the Appendix.
Latent Steps A prominent feature of TAP is that planning happens in a latent action space with temporal abstraction.
Whenever a single latent code is sampled, L steps of transitions that continue the previous trajectory can be decoded.
This design increases the efficiency of planning since the number of steps of unrolling is reduced to 1 L ; therefore, the search space is exponentially reduced in size.
However, it is unclear how such a design affects the decision-making performance of the method.
We thus tested TAP with different latent steps L , namely, the number of steps associated with a latent variable.
As shown in the yellow bars in Figure 3, reducing the latent step L tosignificantly undermines the performance of TAP.
We hypothesize that the performance drop is related to the overfitting of the VQ-VAE since we observe the reduced latent step leads to a higher estimated return and also a higher prediction error; see Appendix L for more details.
On the other hand, we found in Appendix Table 7 that the optimal latent steps vary across tasks.
This indicates a fixed latent step may also be suboptimal.
We believe this shows a direction for future work to further decouple the temporal structure between latent actions and decoded trajectories; see Appendix H for further discussion.
OOD Penalty
By varying the threshold for probability clipping ( β ), we also tested the benefits of having both an OOD penalty and return estimations in the objective function Equation (2).
When β = 10 − 5 , the estimated return term dominates the optimization objective for the reward scale of the tasks considered.
When β = 0 .
5 , the OOD penalty is always activated and encourages the agent to choose the most likely trajectory, namely, imitating the behaviour policy.
In these two cases, the performance of the TAP agent drops 30.2% and 20.2%, respectively.
This experiment demonstrates that solely optimizing the return or prior probability leads to suboptimal performance, so both ingredients are necessary for the objective function.
Nevertheless, besides these extreme values, the performance of TAP is robust to a wide range of choices of β , ranging from 0.002 to 0.1, as shown in Table 4.
Planning Horizon As shown in the blue bars in Figure 3, TAP is not very sensitive to the planning horizon, and achieves a mean score of 71 .
9 (-12.9%) by simply expanding a single latent variable, thus doing 3 steps of planning.
This conclusion might be bound to the tasks, as dense-reward locomotion control may require less long-horizon reasoning than more complex decision-making problems.
An ablation (Hamrick et al., 2021) of MuZero also shows that test-time planning is not as helpful in more reactive games like Atari compared to board games.
Beam Search We also investigate whether structured decoding in the form of beam search is helpful, compared to an alternative strategy of simply autoregressive sampling from the prior policy to generate action proposals.
We test the TAP agent with this form of random shooting by sampling 2048 trajectories from the policy prior.
As shown in the green bar with the ”Prior” label in Figure 3, beam search still performs slightly better.
Moreover, its decision latency is lower because the beam width (64) is much smaller than the number of samples needed for direct sampling (2048).
On the other hand, the fact that even direct sampling can generate decent plans for TAP shows that the latent space for TAP is compact and can be used for efficient planning.
Sampling from a uniform action prior When TAP performs beam search or direct sampling, the latent action codes are sampled from the learned prior policy rather than the uniform distribution.
Such a design is straightforward for a VQ-VAE since the objective, in that case, is to sample from the dataset distribution.
However, in the case of RL, the aim is different since we would like to find an optimal trajectory within the support of the data.
We therefore also test direct sampling where the latent codes are uniformly sampled.
The performance of this approach is illustrated as the green bar with a ”Uniform” label in Figure 3.
Sampling according to the uniform distribution largely damages (-37.9%) the performance of the agent, even with an OOD penalty.
To further investigate the role of sampling from the prior, in Appendix M in we visualize the distribution of trajectories modelled by
TAP in two locomotion tasks.
Unconditional Decoder One of the key design choices of TAP is conditioning the decoder on the initial state of a trajectory.
This allows us to model the (conditional) distribution of trajectories with a very small number of latent variables in order to construct a compact action space for downstream planning.
While it might be obvious that representing multiple steps of high-dimensional states and actions with a single latent variable is difficult, we present an empirical investigation of the performance of TAP with the unconditional decoder.
The orange bars in Figure 3 show that without the first state as input to the decoder, the performance of TAP drops drastically.
This is because, given the same number of latent variables, the reconstruction accuracy of the unconditional decoder is much lower.
Slightly increasing the number of latent variables from L = 3 to L = 1 also does not alleviate the performance drop.
Details about this ablation can be found in the Appendix Table 9.
C R ELATED W ORK
Model-based RL TAP fits into a line of work on model-based RL method (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015; Wang & Ba, 2020; Schrittwieser et al., 2020; Amos et al., 2021; Hafner et al., 2021) because it makes decisions by predicting into the future.
In such approaches, prediction often occurs in the original state space of an MDP, meaning that models take a current state and action as input and return distribution over future states and rewards.
TAP is different from these conventional modelbased methods since the TAP decoder does not strictly follow the MDP causal structure but takes latent variables and the current state as inputs, returning a whole trajectory.
These latent variables act as a learned latent action space which is compact and allows efficient planning.
Planning in the Latent Space Hafner et al.
(2019) and Ozair et al.
(2021) proposed to learn a latent state space and dynamics function in the latent space.
However, in their cases, the action space of the planner is still the same as the raw MDP and the unrolling of the plan is still tied to the original temporal structure of the environment.
Wang et al.
; Yang et al.
(2021) proposed to learn representations of actions on-the-fly for black-box optimization and path planning setting.
While related at a high level, these works operate on environment dynamics that are assumed to be known in advance.
TAP extends this idea to the RL setting where the true dynamics of the environment are unknown, requiring a dynamics model and latent action representation space to be learned jointly.
RL as Sequence Modelling TAP follows a recent line of work that treats RL as a sequence modelling problem (Chen et al., 2021; Janner et al., 2021; Zheng et al., 2022).
Such works have used
GPT-2 (Radford et al., 2019) style Transformers (Vaswani et al., 2017) to model the whole trajectory of states, actions, rewards and values, and turn the prediction ability into a policy.
DT applies an
Upside Down RL (Schmidhuber, 2019) approach to predict actions conditioned on rewards.
TT applies planning in order to find the best trajectory that gives the highest return.
The strong sequence modelling power of the Transformer enables TT to generate long-horizon plans with high accuracy.
However, since planning is in the raw state and action space, the decision latency becomes a bottleneck in high-dimensional environments.
TAP provides a solution for TT and all other discrete space planning algorithms to efficiently plan in complex action spaces.
Offline RL In this paper, TAP is tested in an offline RL setting (Ernst et al., 2005) where we are not allowed to use online experience to improve our policy.
A major challenge of offline RL is to prevent out-of-distribution (OOD) actions to be chosen by the learned policy, so that the inaccuracy of the value function and the model will not be exploited.
Conservatism has been proposed as a solution to this issue (Kumar et al., 2020; Fujimoto & Gu, 2021;
Lu et al., 2022; Kostrikov et al., 2022; Kidambi et al., 2020).
TAP can naturally prevent the OOD actions (or, more generally, OOD sequences) because it models trajectories generated by behaviour policy.
To further discourage OOD trajectories from being selected as a plan, we add an OOD penalty given by the learned prior to the planning objective.
Hierarchical RL In the literature of hierarchical RL, a conceptually relevant work to ours is (CoReyes et al., 2018), which uses a VAE to project the state sequences into continuous latent variables and conditions policies on not only the current state but also the latent variable.
Whereas TAP models actions, states, rewards, and values jointly with a unified Transformer, Co-Reyes et al.trains a state decoder and a step-by-step policy decoder separately.
The policy decoder in their case has to be trained with a conventional RL algorithm with approximate gradients, whereas TAP follows the sequence modelling approach and is trained with fully (self-)supervised signals.
Since the high-level latent actions are discrete, our work is also related to RL methods in the options framework (Sutton et al., 1999; Stolle & Precup, 2002;
Bacon et al., 2017) since both the latent codes of TAP and options introduce a mechanism for temporal abstraction.
Conceptually, a latent action model can fully decouple the planning from the environment’s temporal structure and go beyond the feedforward shooting-based planning (see Appendix H for details).
In TAP, the latent actions emerge from the training of a generative model of trajectories, causing action segments to be grouped into the same discrete latent action if they lead to similar future trajectories.
Such a latent action space is a natural fit for model-based RL and efficient planning.
D S ETUP FOR T RAJECTORY A UTOENCODING P LANNER
We keep most of the design decisions the same as in TT in order to show the advantage of the latentaction model.
Namely, (1) no bootstrapped value estimation is used; (2) the trajectory is treated as a sequence and the latent structure of the model does not follow the structure of the MDP; (3) beam search is used for planning; and (4) the model architectures for the encoder, decoder, and prior policy are Transformers.
As for the TAP-specific hyperparameters: we set the number of the steps associated with each latent variable to be L = 3 and each latent variable has K = 512 candidate values.
The planning horizon in the raw action space is 15 for gym locomotion tasks and 24 for Adroit tasks.
As such, the planning process only needs to optimize in a 5 or 8 -dimensional discrete space with a learned prior.
Other hyperparameters including architectures can be found in the Appendix.
We test each task withtraining seeds, each evaluated for 20 episodes.
Following the evaluation protocol of TT and IQL, we use the v2 version of the datasets for locomotion control and v0 for the other tasks.
E S CALE U P THE P LANNING H ORIZON
TAP not only makes it easier to scale up the state and action dimensionality, but also helps the agent to plan over a longer horizon with much less computation.
As shown in Figure 4(c), the decision latency increases with the number of planning steps, tested on the hopper-medium-replay task.
TAP is able to plan 33 steps into the future within 0.25 seconds.
Table 3 shows how the performance on locomotion control tasks varies with the planning horizon.
We can see that even just expanding a single latent variable (corresponding to 3 steps in the original space), TAP can actually achieve decent performance in most tasks.
The longer horizon planning is helpful for hopper, especially in medium-replay settings.
In hopper tasks, the agent can easily fall over compared to other tasks.
Once the agent falls over, the episode will be immediately terminated, so similar actions might lead to drastic longer-term differences.
This might mean learning the dynamics function and using it to help learn the value function is easier than directly learning the value function.
On the other hand, the medium-replay dataset has a higher diversity which may make longer planning useful.
Both medium and medium expert datasets are generated by one or two fixed policies.
This means the possible actions for the next step are quite limited, so searching may not give very different results given the estimated return.
The medium-replay dataset, on the other hand, contains the data in the replay buffer.
Given the policy parameters are changing during learning, the behaviour policy is then a mixture of a lot of different policies, making TAP has to consider more options when acting.
F T HE R OLE OF OOD P ENALTY
Besides sampling from the prior distribution given by the transformer, we also used the estimated prior probability to prevent out-of-distribution (OOD) trajectories to be selected during the planning.
In Table 4 we showed how the threshold of OOD penalty will affect the performance of TAP.
We can see when a very low threshold is chosen β = 10 − 5 , the performance of the agent drops significantly because the trajectory of such low prior probability will be not sampled in the first place and the OOD penalty is not working.
Some of the OOD trajectories with high prediction error and high value will be selected in this case.
A larger β in the range of [0 . 002 , 0 . 1] gives consistent similar results.
However, keeping improving β to 0 .
5 damages the performance again as most of the trajectories do not have such a high prior probability and the penalty will force the agent to choose the most
[Figure]
[Table]
probable trajectory.
This β
= 0 .
5 case can also be treated as an imitation learning baseline, showing searching for the higher return trajectory rather than cloning the behaviour policy is helpful
[Table]
Environment
[Table]
G B ASELINES
As for the baselines, we gather the strong baselines for each of the three sets of tasks.
We include model-based methods such as MOReL (Kidambi et al., 2020) and (Optimized) MOPO
(Yu et al., 2020; Lu et al., 2022); actor-critic methods CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2022); and sequence modelling methods DT and TT.
We use scores reported by the papers by default with a few exceptions.
• All the methods besides TT did not report performance on Ant locomotion control so we run them using their official codebase.
• For AntMaze-Ultra, we run IQL with their official codebase and run TT (+ Q ) with the code given by the author (Janner et al., 2021).
• The Ant locomotion results of TT come from their official GitHub repository 2 .
• Since CQL is tested in the v0 version of locomotion control tasks, we use the v2 results reported by IQL (Kostrikov et al., 2022).
• All the behaviour cloning (BC) results are reported by D4RL (Fu et al., 2020).
• Chen et al.
(2021, DT) did not report the results for AntMaze and Adroit so we use DT AntMaze results reported by IQL (Kostrikov et al., 2022).
• MOPO results for Adroit come from (Lu et al., 2022).
• TT results for adroit come from (Wang et al., 2022).
We also tested two training seeds, each 10 evaluation episodes, for pen-human and pen-cloned and get 12 .
9 ± 9 . 2 and 17 .
3 ± 9 .respectively.
H O THER D ESIGN P OTENTIALS FOR THE B OTTLENECK
The design of the prior model is highly conditioned on the design of the bottleneck and the decoder, and will have an impact when used for planning.
Following Decision Transformer (Chen et al., 2021) and Trajectory Transformer (Janner et al., 2021), we used a GPT-2 style transformer with causal masking for our encoder and decoder.
In this case, the information in the future token will not follow back to recent ones.
Such a design is conventional for sequence modelling but not necessarily optimal.
For example, one could reverse the order of the masking for the decoder and make the planning goal-oriented, whereas the prior should also be trained in a reversed order.
To fully decouple the planning and temporal structure, we also tried using attention rather than pooling and tiling to construct the bottleneck.
Such a design shows a similar performance when doing vanilla sampling.
Such a design also prevents us from applying beam search and is thus less efficient and we did not put that to the main text.
However, the fact that TAP with attention bottleneck works as well as fixed length step length indicates the temporal structure of the MDP may not be a necessary inductive bias for planning.
This can motivate future works that design more flexible and efficient ways of planning based on the latent-action model.
I A NT M AZE D ETAILS
AntMaze is a sparse-reward continuous-control task in which the agent has to control a robotic ant to navigate to the target position.
We include the goal position in the observation space so that the generated trajectories are goal-aware.
In order to extensively test different methods, we also add a customized larger antmaze that we called Antmaze-Ultra, which has doubled the size of the previously largest map (Antmaze-Large).
Antmaze has the same dimensionality as locomotion ant and so will not provide too much direct evidence about the action dimension scalability, so it’s more about an orthogonal evaluation on sparse reward problems.
AntMaze is challenging because there are a lot of sub-optimal trajectories in the dataset that is navigating to different goal positions rather than the target position in the test time.
Reaching these goals will not give a reward to the agent.
The reward will only be given when the agent reaches the true target, which is also the goal in the test time.
On AntMaze, TAP shows better performance on a more challenging large dataset, but is slightly inferior to TT (+ Q ) on the medium datasets.
We did not use the IQL critic as our value estimator; a better value estimation approach than our current Monte-Carlo estimation would be expected to bring orthogonal improvements.
Effective performance without a separate Q -function is partially due to the inclusion of the goal position into the observation: TAP can learn to generalize across goal positions instead of completely relying on the value estimation.
TT has shown using a separate IQL value function can help sequence generative models to solve
AntMaze.
Such an approach, however, further increases the computational cost for sampling trajectories.
Here we instead propose an alternative efficient approach that let TAP jointly leverage trajectories with different objectives and also the reward signals.
This is as simple as putting the goal positions to the observation of the agent.
We hypothesized that goal positions can be helpful for TAP as it models the whole distribution of the trajectories.
When acting in the environment, having the Trajectories conditioned on the goal can narrow down the sampled trajectories to focus on the correct direction, therefore simplifying the planning.
Note that for actor-critic methods, the positions of the test time target position are encoded by the critic network and including the goal position will not provide extra information to the agent.
In order to pressure test IQL, TT + Q and TAP in larger AntMaze environments, we introduced the
AntMaze-Ultra task, which has 10 × 14 blocks of effective size.
This is 4 and 2 times as large as AntMaze-Medium ( 6 × 6 ) and AntMaze-Large ( 7 × 7 ) tasks, respectively.
A visualisation of these three tasks can be found in Figure 5.
In addition to the increased size, the complexity of the walls also results in multiple possible routes to navigate from the bottom left corner to the top right.
Similar to the antmaze-medium and antmaze-large tasks, we let the agent run for 1K episodes, each with 1K steps to collect the trajectories.
However, we find that such an amount of data is far away from being enough to learn a proper value function, as IQL gives a very poor performance on this task.
Using the Monte-Carlo estimation, TAP even suffers more from this issue.
We find that planning with beam search, in this case, is even worse than just sampling a random trajectory: with random sampling, TAP gives 22.00 +/- 4.14 on AntMaze-Ultra-Play and 26.00 +/- 4.39 on
AntMaze-Ultra-Diverse.
However, beam search will decrease its performance to 21.00 +/- 4.07 on
Ultra-Play and 22.00 +/- 4.14 on Ultra-Diverse.
[Table]
J I NTERPRET L ATENT C ODES
A key design choice we made is to let the decoder, not only the prior of latents, be conditioned on the current state.
Such a design leads to a compact latent action space that enables efficient planning.
Without the state as input, the decoder must reconstruct the whole trajectory relying solely on the latent codes, so the latent codes contain information about the whole trajectory segments (states, actions, rewards, values).
In other words, latent codes will be a discrete representation of the trajectory, where a latent codes sequence can be decoded into a unique trajectory.
On the other hand, conditioned on the current state, the latent codes only need to carry information about how the future trajectories branch from the existing trajectory.
Every L step, a branch of the trajectory will fork into K possible futures.
Each of the latent codes corresponds to a branch of this tree of possible future trajectories, where the root node is the current state.
The forking can be caused by the uncertainty of the dynamics model or by the stochasticity of the behaviour policy.
The number of
[Figure]
latent codes needed to reconstruct a trajectory given a starting state will reduce as long as the model knows more about the dynamics of the environment and the behaviour policy.
Assume a trajectory distribution p ( τ | sss 1 ; µ ) with behaviour policy µ has deterministic Markovian transition dynamics, and the model has recovered the true dynamics.
The probability of a valid trajectory that fit the dynamics can be expressed as p ( τ | sss 1 ; µ ) =
(cid:81)
Ti =1 µ ( aaa i | sss i ) .
So the discrete latent variables only need to approximate the distribution of the behaviour policy, where latent codes correspond to grouped components of the behaviour policy.
Note that the policy is not only grouped in a single step but also across multiple steps.
Intuitively, the latent codes can then be interpreted as state-conditioned options (Sutton et al., 1999), learning by decomposing the behaviour policy into a weighted sum of sub-policies.
The number of grouped policy components of the behaviour policy can be much less than the dimensionalities needed to represent the whole trajectory.
As we will show in Appendix B, the L = 3 or L = 4 gives an optimal performance of TAP on D4RL tasks, which means 3 steps of the transitions with up to hundreds of dimensions can be expressed with a single discrete latent variable, with K = 512 bins.
In contrast, the discrete representation of the trajectory used in TT will allocate LD discrete codes to describe the same number of transitions.
This means TAP has a smaller and more compact latent space for downstream planning.
K H YPERPARAMETERS
In Table 6 we showed the hyperparameters for TAP, both for training and planning.
L P REDICTION ERROR , SEARCH VALUE AND P ERFORMANCE
To investigate the reason that temporal abstraction and state-conditional encoder will be helpful for
TAP, we provide more metrics about TAP search in Figure 6.
Scores show the performance of the agent.
Search values are optimal value(return) found by TAP in the initial states.
Errors are mean squared errors of state prediction given the action sequences output by the TAP.
We can see: 1)
[Table] [Table]
For state-conditional decoder, there is a negative correlation between prediction errors and scores.
This not surprising because better prediction accuracy can naturally lead to better performance.
For both state-conditional and unconditional decoder, search values are positively correlated to the errors.
This indicates extra value improvement might be caused by over-optimistic prediction. 3)
Both search value and errors are minimal at L = 3 .
Our interpretation about this is: Smaller L can lead to overfitting in the training data, therefore worse generalization and finer-grained search, which will amplify generalization error.
On the other hand, larger L can cause underfitting therefore
[Figure]
higher error.
Naively increasing the number of latent variables is not helpful for the unconditional decoder.
[Table]
[Table]
M S AMPLED D ISTRIBUTIONS
We sampled 2048 trajectories either uniformly or according to the prior in the first step of theevaluation episodes and plot their metrics in Figure 7.
The x-axis of the plots is predicted returns
[Figure]
by the agents.
The y-axis shows the state’s prediction mean-squared error (MSE), where the ground truth is given by the simulator following the action sequences of the predicted trajectories.
Sampling from the prior does not damage the diversity of the sampled trajectories in terms of the variance of returns.
On the other hand, the expected MSE is reduced in both hopper and ant cases.
This means that by sampling from the prior, we can get a higher quality of trajectory samples.
We can see that TAP successfully approximates the continuous distribution of returns and the generated trajectories are high in diversity in terms of the returns.
This is especially true for hopper-mediumreplay because the behaviour policy itself is of high diversity, and we can see multiple modes in the distribution.
It is interesting to see that the trajectories with higher predicted returns do not necessarily have higher prediction errors.
This is a nice feature because we can search for highreturn trajectories without worrying too much about these trajectories being unrealistic.
In the ablation of sampling from prior versus sampling from a uniform distribution, we saw sampling from the prior policy gives much better performance.
Comparing the distribution of returns and errors of trajectories sampled from the prior and uniform distribution, we observe the sampling from prior policy yields trajectories with lower reconstruction error but with similar returns.
This indicates sampling from prior will improve the quality of trajectories being generated from TAP, which explains better performance.
N T RAINING C OST
The training time of TAP and TT are similar for lower dimensionality, TT needs 6-12 hours and
TAP needs 6 hours for gym locomotion control tasks, on a single GPU.
It’s worth noting that the training cost of TT will also grow quickly along the dimensionality.
For adroit, the same training for
TT takes 31 hours, but the training cost of TAP is the same given the same architecture.
In fact, for adroit experiments with TAP, we used a smaller network (see Table 6 for hyper-parameters) which needs fewer training epochs and the whole training can be finished within 1 hour.