Paper: Learning to Act without Actions (ICLR-2024)

Reviewer comment (Experiments):
Toy experimental setup: The paper motivates from learning (pretraining) with large scale web data. However, during experiments, the author uses the data from a simulator (Procgen). The pretrain data contains less visual complexity compared to real world data. I encourage author to explore pretraining with real world data (like MC YouTube videos [1] and first-view driving videos [2]). Or, the authors should at least make the pretraining data different from the RL observation data to simulate the difference between pretrain and finetune scenario. One way could be using different backgrounds in Procgen.

Author response:
We generally agree and intend to address the evaluation of LAPO against real-world observational data in future work. However we believe that our use of a synthetic benchmark does not reduce the utility or novelty of our work. Similar criticism could be leveled against dozens of papers that used Procgen as an evaluation benchmark, many of which were instrumental in developing the current state-of-the-art methods in RL. Indeed, Procgen was developed to be a useful evaluation setting for developing new methods in deep RL and is an effective tool to rapidly perform experiments in a tightly controlled setting. Likewise, the VPT work itself is based on a video-game environment that only loosely resembles the real world.

Reviewer comment (Novelty):
In my opinoin, this paper could benefit from including references to a relevant work [2] which uses IDM to label the dataset and conduct contrastive learning for policy pretraining.

Author response:
Thank you for this suggestion. We agree that this is relevant prior work and included it in the related works section of the updated paper.

Reviewer comment (Experiments):
I think this paper could benefit from additional comparison with [1] and [2]. It could provide additional insight of how methods that use extra data for IDM training perform compared to the proposed method. I am particularly interested in understanding whether training IDM yields performance improvements or if the proposed method (without IDM) is already capable of achieving comparable, or even superior, results to the aforementioned prior works [1][2].

Author response:
While VPT `[1]`, ACO `[2]`, and LAPO seem to tackle a similar problem on the surface, we want to strongly emphasize that the intention behind the design of LAPO is fundamentally different. Our primary design consideration with LAPO was to push the need for true action labels as far back in the pretraining/finetuning pipeline as possible. While other methods like VPT, ACO, SS-ORL `[3]`, and RLV `[4]` need action labels to even train an IDM, not to mention a policy, LAPO can train both of these, as well as an FDM, with no need for action labels.

We believe this to be a very useful feature of LAPO going forward, as it could allow us to train large-scale general-purpose latent policies on vast amounts of data from the web for which truly no action-labeled data (or even an RL environment) exists and then decode these policies to a true action space using a comparatively minute amount of labeled data from potentially only a small subset of the tasks present in the pretraining data.

`[1]` Baker et al. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. NeurIPS 2022.  
`[2]` Zhang et al. Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining. ECCV 2022.  
`[3]` Zheng et al. Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories. ICML 2023.  
`[4]` Schmeckpeper et al. Reinforcement Learning with Videos: Combining Offline Observations with Interaction. CoRL 2020.

Reviewer comment (Experiments):
Do you think this method can generalize to domains with continuous action space?

Author response:
**We updated the paper to include results on two `dm_control` tasks — please see Figure 9 in the Appendix.** We note that the performance characteristics of LAPO in this setting are similar to our earlier Procgen results, as **LAPO significantly outperforms the PPO-from-scratch baseline.** We aim to provide results for additional continuous control environments in the camera-ready version.

More generally, we note that the latent action space in LAPO is used to describe observed changes from one observation to the next and is thus actually more closely related to the task’s observation space than action space. In Procgen, where we demonstrate the success of our method, the observation space is continuous.

We would also like to highlight that many prior works have successfully used discrete action representations in the context of continuous control. For example, `[1]` and `[2]` use manual, per-dimension quantization schemes for actions and `[3]` goes further by replacing Gaussian policies with Bernoulli policies, in a setting called bang-bang control. Several works learn to quantize continuous actions via vector-quantization, including `[4]` which learns a discrete action representation for use in downstream offline-RL and `[5]`, which applies vector quantization to high-level latent actions. Furthermore, `[6]` learns a quantization scheme from action-labeled demonstrations.


In summary, we are confident that the approach taken by LAPO can be applied to many environments featuring continuous action spaces.

`[1]` Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. NeurIPS 2021.  
`[2]` Tang et al. Discretizing Continuous Action Space for On-Policy Optimization. AAAI 2020.  
`[3]` Seyde et al. Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies. NeurIPS 2021.  
`[4]` Luo et al. Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning. CoRR 2023.  
`[5]` Jiang et al. Efficient Planning in a Compact Latent Action Space. ICLR 2023.  
`[6]` Dadashi et al. Continuous Control with Action Quantization from Demonstrations. ICML 2022.

Reviewer comment (Experiments):
Does the number of discrete tokens in VQ matter? I would appreciate a ablation on this.

Author response:
Yes, the number of discrete tokens that the latent policy generates does matter. When the number of tokens is large, the outcome is similar the No-VQ ablation in the paper (see Figure 3 and Figure 7). This is because the IDM is then able to pass large amounts of information to the FDM and does not need to learn to compress the transition information, thereby effectively negating the bottlenecking effect of vector quantization. When the number of tokens is too small, the performance decreases too. This is because the IDM is then no longer able to pass sufficient information to the FDM. This is particularly noticeable in environments with partial observability, such as Procgen tasks where the camera moves and shows parts of the game in future frames that weren’t visible in the past since in these environments there is a large amount of information that needs to be passed to the FDM.

Reviewer comment (Novelty):
The utility of learned latent representations for large-scale pretraining: It is evident from the plots that the actions are meaningfully clustered. However, if one has enough compute, to me, it seems plausible to train a transformer autoregressively to generate the next observations in observation-only data, ensuring that the latent representation for actions is also learned. I raise this point because if such a pretraining is possible directly with transformers, the utility of LAPO reduces. Plus, if someone wants to apply the LAPO (IDM + FDM) approach, it could be inflexible compared to a transformer extension. Anyway, I do acknowledge that LAPO does provide evidence that it is possible to meaningfully identify actions in observation-only data.

Author response:
Our experiments used simple non-recurrent architectures to represent the IDM, FDM, and policy, since these were entirely sufficient to achieve good results in Procgen and demonstrate the usefulness of our method. However, the ideas behind LAPO are agnostic to the specific architectures used. In more complex environments it could very well be advantageous to implement these models with a large transformer. Referring to your statement "it seems plausible to train a transformer autoregressively to generate the next observations in observation-only data, **ensuring that the latent representation for actions is also learned**", we thus argue that LAPO is *exactly* a solution to this problem. Given a transformer that predicts future observations based on past observations, we would still require some additional method to extract meaningful latent actions from it. LAPO is such a method.

We strongly believe that a transformer-based IDM and FDM could allow LAPO to scale to much more difficult tasks by allowing it to model complex temporal dependencies. Thus, we fully agree scaling LAPO to more powerful, transformer-based models is a promising direction for future work.

Reviewer comment (Experiments):
Issue with NOOP: From the IDM-FDM perspective, the NOOP being a null action, its latent representation should produce no transition when sent to FDM. It was unclear from the writing if there is any experimental validation of the same. Also, there is an issue with NOOP and actions with delayed effects. The present IDM-FDM model will fall short in modeling the two separately. (Authors briefly touch up on modeling the delayed effect actions, but I did not find their mention contrasting them with NOOP's.)

Author response:
We agree with your point 2 in the weaknesses section, that from the perspective of the IDM & FDM, the NOOP action is simply an action like any other. However **we disagree that the NOOP action should produce no transition when sent to the FDM and argue that the NOOP action’s effect depends on the environment.** In Procgen "starpilot" for example, the NOOP action causes the agent to continue moving across the screen at the current velocity. In other Procgen tasks, the physics of the environment as well as the enemy characters similarly continue to affect the state of the environment, even under the NOOP action. Empirically, we observe that the NOOP action is modeled as expected and generally does not affect the transition in tasks such as "maze" and "miner", while indeed affecting the transition in the expected way in tasks like "starpilot". Please see the answer to the next question for evidence of the latent clustering behavior of the NOOP action.

Yes, similar to other actions, the NOOP action also takes on distinct clusters in the latent action space. However, it is important to note that in Procgen tasks, there are actually three kinds of actions that can act as a NOOP action:

1. The actual NOOP action
2. Any action that is invalid in the given environment. For example, the task "maze" only supports the actions LEFT, RIGHT, UP, and DOWN. Procgen internally maps all other actions to NOOP.
3. Any action that is invalid/impossible to perform in a specific state & environment. For example if the agent has a wall immediately to its right, performing the "move right" action is identical to performing the NOOP action.

From the perspective of our action-free IDM & FDM, these three kinds of actions are indistinguishable from one another and therefore end up in the same latent action cluster. Our reason for omitting the NOOP action clusters in Figures 5. and 7. was simply visual clarity, since the NOOP clusters are not color-coded with a single true action label, but rather with all action labels corresponding to the cases 1-3 above.

Note that this clustering behavior also highlights the strength of LAPO’s learned latent action representations, which tends to cluster actions that are more closely related in actual effect on the environment, without relying on the externally-defined action space, which might contain many redundant or equivalent actions.

**Please see Figure 8 in the Appendix of the updated paper for UMAP projection plots that also include the NOOP action.**

The latent action space in LAPO models the *observable effects* of the agent’s actions, rather than the actions themselves. In this way, latent actions can be thought of as highly-compressed descriptions of observed transitions.

In the case of an action $a$ that is taken by the agent at time $t$, but only generates a visible effect at a later time $t+k$, the IDM will be unaware of the action being taken at time $t$ and will only observe the delayed effect. Since at time $t$, true actions $a$ and NOOP are indistinguishable from the IDM’s point of view, the latent action at that time will be the same. As the latent policy is trained to imitate the IDM’s predictions, the policy will generate the NOOP latent action at time $t$, and a latent action describing the visible effect at time $t+1$. We don’t believe this to be a significant issue, since in most environments, actions that have any impact on the state of the environment will usually elicit some degree of immediate change in the observation.

Reviewer comment (Experiments):
LAPO with continuous action environments: The current implementation of LAPO involves using vector quantization (VQ). However, using VQ would, in principle, limit the actions to be chosen from a finite set of discrete codes. This, in turn, creates issues while scaling to continuous action environments. The ProcGen games, environments used in the experiments, are discrete action environments, too, and I find it difficult to see how the approach would scale to non-discrete real-world control tasks.

Author response:
**We updated the paper to include results on two `dm_control` tasks—please see Figure 9 in the Appendix.** We note that the performance characteristics of LAPO in this setting are similar to our earlier Procgen results, as **LAPO significantly outperforms the PPO-from-scratch baseline**. We aim to provide results for additional continuous control environments in the camera-ready version.

More generally, we note that the latent action space in LAPO is used to describe observed changes from one observation to the next and is thus actually more closely related to the task’s observation space than action space. In Procgen, where we demonstrate the success of our method, the observation space is continuous.

We would also like to highlight that many prior works have successfully used discrete action representations in the context of continuous control. For example, `[1]` and `[2]` use manual, per-dimension quantization schemes for actions and `[3]` goes further by replacing Gaussian policies with Bernoulli policies, in a setting called bang-bang control. Several works learn to quantize continuous actions via vector-quantization, including `[4]` which learns a discrete action representation for use in downstream offline-RL and `[5]`, which applies vector quantization to high-level latent actions. Furthermore, `[6]` learns a quantization scheme from action-labeled demonstrations.

In summary, we are confident that the approach taken by LAPO can be applied to many environments featuring continuous action spaces.

`[1]` Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. NeurIPS 2021.  
`[2]` Tang et al. Discretizing Continuous Action Space for On-Policy Optimization. AAAI 2020.  
`[3]` Seyde et al. Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies. NeurIPS 2021.  
`[4]` Luo et al. Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning. CoRR 2023.  
`[5]` Jiang et al. Efficient Planning in a Compact Latent Action Space. ICLR 2023.  
`[6]` Dadashi et al. Continuous Control with Action Quantization from Demonstrations. ICML 2022.

Reviewer comment (Experiments):
The approach is only evaluated in one environment, Procgen. It is unclear if this approach will generalize across different domains (i.e. continuous control/Robotics). Additional experiments in other environments could strengthen the case of the paper.

Author response:
Dear Reviewer Z13u,  
We sincerely thank you for your valuable feedback on our paper. We particularly appreciate your support of the strength of our experiments. We address your question below.

> Do the results of LAPO generalize to other environments, especially in continuous control settings? (such as MoJoCo, DMControl, Meta-World, etc.)

**We updated the paper to include results on two `dm_control` tasks—please see Figure 9 in the Appendix.** We note that the performance characteristics of LAPO in this setting are similar to our earlier Procgen results, as LAPO significantly outperforms the PPO-from-scratch baseline. We aim to provide results for additional continuous control environments in the camera-ready version.

More generally, we note that the latent action space in LAPO is used to describe observed changes from one observation to the next and is thus actually more closely related to the task’s observation space than action space. In Procgen, where we demonstrate the success of our method, the observation space is continuous.

We would also like to highlight that many prior works have successfully used discrete action representations in the context of continuous control. For example, `[1]` and `[2]` use manual, per-dimension quantization schemes for actions and `[3]` goes further by replacing Gaussian policies with Bernoulli policies, in a setting called bang-bang control. Several works learn to quantize continuous actions via vector-quantization, including `[4]` which learns a discrete action representation for use in downstream offline-RL and `[5]`, which applies vector quantization to high-level latent actions. Furthermore, `[6]` learns a quantization scheme from action-labeled demonstrations.

In summary, we are confident that the approach taken by LAPO can be applied to many environments featuring continuous action spaces.

`[1]` Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. NeurIPS 2021.  
`[2]` Tang et al. Discretizing Continuous Action Space for On-Policy Optimization. AAAI 2020.  
`[3]` Seyde et al. Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies. NeurIPS 2021.  
`[4]` Luo et al. Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning. CoRR 2023.  
`[5]` Jiang et al. Efficient Planning in a Compact Latent Action Space. ICLR 2023.  
`[6]` Dadashi et al. Continuous Control with Action Quantization from Demonstrations. ICML 2022.

Reviewer comment (Experiments):
Do the results of LAPO generalize to other environments, especially in continuous control settings? (such as MoJoCo, DMControl, Meta-World, etc.)

Author response:
Dear Reviewer Z13u,  
We sincerely thank you for your valuable feedback on our paper. We particularly appreciate your support of the strength of our experiments. We address your question below.

> Do the results of LAPO generalize to other environments, especially in continuous control settings? (such as MoJoCo, DMControl, Meta-World, etc.)

**We updated the paper to include results on two `dm_control` tasks—please see Figure 9 in the Appendix.** We note that the performance characteristics of LAPO in this setting are similar to our earlier Procgen results, as LAPO significantly outperforms the PPO-from-scratch baseline. We aim to provide results for additional continuous control environments in the camera-ready version.

More generally, we note that the latent action space in LAPO is used to describe observed changes from one observation to the next and is thus actually more closely related to the task’s observation space than action space. In Procgen, where we demonstrate the success of our method, the observation space is continuous.

We would also like to highlight that many prior works have successfully used discrete action representations in the context of continuous control. For example, `[1]` and `[2]` use manual, per-dimension quantization schemes for actions and `[3]` goes further by replacing Gaussian policies with Bernoulli policies, in a setting called bang-bang control. Several works learn to quantize continuous actions via vector-quantization, including `[4]` which learns a discrete action representation for use in downstream offline-RL and `[5]`, which applies vector quantization to high-level latent actions. Furthermore, `[6]` learns a quantization scheme from action-labeled demonstrations.

In summary, we are confident that the approach taken by LAPO can be applied to many environments featuring continuous action spaces.

`[1]` Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. NeurIPS 2021.  
`[2]` Tang et al. Discretizing Continuous Action Space for On-Policy Optimization. AAAI 2020.  
`[3]` Seyde et al. Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies. NeurIPS 2021.  
`[4]` Luo et al. Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning. CoRR 2023.  
`[5]` Jiang et al. Efficient Planning in a Compact Latent Action Space. ICLR 2023.  
`[6]` Dadashi et al. Continuous Control with Action Quantization from Demonstrations. ICML 2022.

Reviewer comment (Experiments):
The experiments assume that the underlying action space in the videos used for pre-training is the same as the action space of the agent. In other words, the videos used during pre-training are of the same agent. I think this assumption is limiting. To leverage large-scale video data available on the internet, one of the key ingredients is learning from videos that doesn’t necessarily match the action space of the agent. Imagine learning from Ego4D like ego-centric observations, and using it to execute tabletop rearrangement tasks. Or using RealEstate10K videos to learn how to navigate in indoor environments.

Author response:
We agree that being able to leverage data from a variety of domains and tasks is crucially important for learning from large-scale video data. While we believe this to be out of scope for this current paper, we are confident that our method could be well applicable to this multi-task transfer setting. **During the IDM/FDM and latent behavior cloning training stages we make no strong assumptions on the true action space of the data-generating policy.** Even if the action space (and possibly observation space) of the downstream RL task is different from the environment of the data-generating policy, the learned latent policy might still capture information useful for tackling this different environment. We believe this multi-task setting to be a promising line of research for future work extending LAPO.

Reviewer comment (Experiments):
It would have also been nice to consider experiments on continuous control tasks. The approach uses a vector quantised latent space to model actions. Will such an approach also work for actions that are continuous control?

Author response:
Thank you for this suggestion. **We updated the paper to include results on two `dm_control` tasks—please see Figure 9 in the Appendix.** We note that the performance characteristics of LAPO in this setting are similar to our earlier Procgen results, as LAPO significantly outperforms the PPO-from-scratch baseline. We aim to provide results for additional continuous control environments in the camera-ready version.

More generally, we note that the latent action space in LAPO is used to describe observed changes from one observation to the next and is thus actually more closely related to the task’s observation space than action space. In Procgen, where we demonstrate the success of our method, the observation space is continuous.

We would also like to highlight that many prior works have successfully used discrete action representations in the context of continuous control. For example, `[1]` and `[2]` use manual, per-dimension quantization schemes for actions and `[3]` goes further by replacing Gaussian policies with Bernoulli policies, in a setting called bang-bang control. Several works learn to quantize continuous actions via vector-quantization, including `[4]` which learns a discrete action representation for use in downstream offline-RL and `[5]`, which applies vector quantization to high-level latent actions. Furthermore, `[6]` learns a quantization scheme from action-labeled demonstrations.


In summary, we are confident that the approach taken by LAPO can be applied to many environments featuring continuous action spaces.

`[1]` Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. NeurIPS 2021.  
`[2]` Tang et al. Discretizing Continuous Action Space for On-Policy Optimization. AAAI 2020.  
`[3]` Seyde et al. Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies. NeurIPS 2021.  
`[4]` Luo et al. Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning. CoRR 2023.  
`[5]` Jiang et al. Efficient Planning in a Compact Latent Action Space. ICLR 2023.  
`[6]` Dadashi et al. Continuous Control with Action Quantization from Demonstrations. ICML 2022.