Paper: ALP: Action-Aware Embodied Learning for Perception (ICLR-2024)

Reviewer comment (Evaluation):
The proposed method underperforms compared to SimCLR ImageNet baseline in table 2 on train split and the results for depth estimation are not provided which contradicts the claims made by the authors.

Author response:
**Explanation of RND-ALP performance**

For SimCLR (FrImgNet) in Table 2, we initialize with ImageNet SimCLR pre-trained weights, and continue to learn representation from all actively explored frames by RND-ALP using contrastive loss with data augmentation. Our intuition is that ImageNet SimCLR provides a stronger initialization since it sees much more diverse curated data, and together with actively explored diverse samples from the simulator this boosts the representation quality in training environments, while at the same time we generally found that contrastive loss slightly hurts performance in unseen test scenarios. However, we found that ALP outperforms other visual representations trained from-scratch in simulator (Table 2 and 7); this shows action aware training objectives result in performance gains than self-supervised contrastive learning in embodied settings.

Reviewer comment (Evaluation):
As shown in table 3 the results of finetuning different baselines using same finetuning dataset (collected using RND-ALP) the proposed method is only slightly better ~0.2-0.7% compared to ImageNet baseline. It’d be nice if authors run this experiment with multiple seeds and share mean and variance to show the improvements are consistent across seeds.

Author response:
**Results from multiple random seeds**

We already include experimental results of multiple random seeds for RND-ALP in Table 11 in Appendix. Given the same pre-trained representation and the same task samples dataset, our 3 runs of results achieve similar performance.

We additionally include downstream performance of RND-ALP over 2 runs with different runs for both pre-training and fine-tuning stages below. We generally found that performance in unseen scenarios contains slightly higher variance than that in training environments across multiple runs; overall our method could achieve consistent performance.

| Train Split || Test Split ||
| :--- | :--- | :--- | :--- |
| ObjDet | InstSeg | ObjDet | InstSeg | 
| 87.95 | 84.56 | 50.34 | 46.74 |
| 88.29 | 83.60 | 50.82 | 43.20 |

Reviewer comment (Experiments):
Table 4 compares different data collection methods for finetuning where RND-ALP outperforms existing methods. I’d be interested to see a comparison with a baseline that collects data using random sampling strategy and using a heuristic based exploration agent like a frontier exploration agent which is maximizing the coverage in the environment. The frontier agent baseline is similar to ANS but builds a map using depth + camera info which leads to much better maps for exploration. This can be implement by leveraging codebase from [1]. Based on videos attached in supplementary it looks like RND/CRL/RND-ALP are learning simple navigation behavior which might restrict the diversity of the dataset used for training. It’d be nice to compare the performance with such simple heuristic based data collection baselines.

Author response:
**Data collection baselines**

We report experimental results of fine-tuning different visual representations in downstream labeled data using random policy in Table 8 of Appendix C. We also present further comparisons with ANS method in the same Table, which tends to achieve stronger performance than frontier-based exploration. We generally found that models trained on passive samples from random policy underperform models trained on active data in both evaluation settings. This demonstrates the importance of active data collection in improving visual representation learning and training better downstream task models.

Reviewer comment (Experiments):
In addition to ImageNet pretrained baselines there has been some recent works like MAE, VC-1[2], R3M[3], MVP[4]. Can authors please explain why the comparison with these representation learning baselines are missing? I'd like to see how well ALP does in comparison to these other pretrained representations.

Author response:
**Visual representation learning baselines**

We compare to image-level visual pre-training baselines on fixed and static datasets such as ImageNet, since we propose to incorporate action signals from active embodied movements into learning visual representation in our ALP framework, which is not available when pre-training on fixed and static datasets. Given that contrastive loss is a popular self-supervised learning approach and can be integrated with a wide range of model architectures, we mainly compare to ImageNet SimCLR as the representation learning baseline model pre-trained on large-scale curated datasets. We additionally compare to CRL [1] as the previous embodied learning baseline method, and refer to ImageNet supervised pre-training in Table 3. R3M [3] utilizes language annotations of egocentric videos as another data modality that are not available in our settings, and focuses on improving success of robotics manipulation tasks, differently from our goals; thus we did not include it as a baseline in our paper draft.

Importantly, we want to show that by actively exploring in embodied environments the agent can effectively learn visual representations from action information. *We are not trying to present a better pre-trained visual representation model at this point;* instead we want to deliver our key insight that leveraging learning signals from active embodied movements can provide substantial gains in learning visual perception, as also shown in Table 2 and 7.

Given that more recent pre-trained visual representations are proposed, we provide reference to performance of these suggested models, including MAE [4], VC-1 [5], and R3M [3], below. We initialize the backbone encoder of Mask-RCNN models with each of pre-trained visual representations, and fine-tune on the same downstream task dataset collected from RND-ALP. We use default training configurations as ViTDet in detectron2 library [6].

| | | Train split || Test Split ||
| :--- | :--- | :--- | :--- | :--- | :--- |
| Method | Model Architecture | ObjDet | InstSeg | ObjDet | InstSeg |
| MAE | ViTDet w/ ViT-B | 92.09 | 88.32 | 60.26 | 53.87 |
| VC-1 | ViTDet w/ ViT-B | 91.72 | 87.61 | 58.84 | 54.44 |
| R3M | ResNet-50 w/ FPN | 88.66 | 84.71 | 32.27 | 26.86 |
| RND-ALP | ResNet-50 w/ FPN | 87.95 | 84.56 | 50.34 | 46.74 |

*However, our goal is not to outperform these methods here. Datasets used to train these representations, such as ImageNet and Ego4D, require large amounts of human effort for collections and annotations. This is different from our framework, where pre-training and fine-tuning datasets are autonomously collected from an active exploration agent moving with intrinsic motivations.* Specifically, R3M is trained with a video-language alignment loss on egocentric manipulation videos, which may not be especially suitable for visual semantic tasks; although MAE and VC-1 achieves much better performance, it is commonly known that ViTDet significantly outperforms on COCO benchmark [6, 7]. Again these models are trained on curated internet-scale datasets that are manually collected and annotated with huge effort. Thus, we don’t think missing comparisons to these visual representations in our paper draft would be a weakness of our framework.

Reviewer comment (Experiments):
For each of the tasks ObjDet, InstSeg, and DepthEst can authors also add comparison with the SOTA baselines for each task finetuned on data collected for finteuning? For example, it is important to compare how well a coco pretrained ObjDet model when finetuned on sim data for the 6 categories being used.

Author response:
**Compare to COCO pre-trained models**

We provide experiment results of fine-tuning COCO pre-trained models below. All of these models are trained on the same task samples dataset collected from RND-ALP. We found that in our settings, initializing with ImageNet supervised pre-training backbone and COCO pre-trained Mask-RCNN generally achieved similar performance.

| | Train split || Test Split ||
| :--- | :--- | :--- | :--- | :--- |
| Method | ObjDet | InstSeg | ObjDet | InstSeg |
| RND-ALP (ours) | 87.95 | 84.56 | 50.34 | 46.74 |
| ImageNet Supervised | 88.75 | 85.51 | 52.78 | 46.94 |
| COCO Supervised | 88.57 | 84.35 | 52.64 | 47.81 |

Again our goal is not to outperform the performance of these models here, since their pre-training has access to a significant amount of supervisions from semantic class labels or annotations and constructing these curated datasets require huge manual efforts.

Reviewer comment (Experiments):
The scope of experiments is only limited to simulated environments of 3D scans of real locations (the Gibson environments).

Author response:
While we perform all experiments in the Gibson dataset [1], different scenes in the Gibson dataset contain some extent of diversity. We follow prior work [2] in splitting training and unseen test scenes, details provided in Appendix B2. Our evaluations on unseen test scenarios could indicate some generalization capability of our framework, since these environments are not available during either pre-training or fine-tuning stages. We found it a little challenging to directly evaluate on another dataset such as Replica or Matterport3D, since these datasets capture different styles of scenes or buildings and distributions of object categories are different, which could add further complications to evaluations. As discussed in the last part, we would like to leave further extensions to other simulator datasets and/or real world environments as part of future work.

Reviewer comment (Experiments):
The action space utilized in the experiments is relatively simple, including only discrete actions, with no further experiments on more complex action space.

Author response:
We define the action space in our experiments same as in prior works [2]. We render the Gibson dataset in the Habitat simulator [4] and train a reinforcement learning policy of a navigation agent. We tried looking up and down as possible actions similar to CRL [3], but we found that this collects weaker samples, e.g. the agent might get many observations of ceilings and floors without valid semantic annotations of objects. We thus use the same action space as SEAL [2]. 
We are not aware of different choices of embodied agents as you described under our setup; could you further clarify this part? Thanks! Although we currently utilize a simple action space in our experiments, we already observe performance gains, indicating effectiveness of leveraging action information for visual representation learning. As discussed in the last part, we would like to leave further investigations on more complex action space as part of future work.

Reviewer comment (Novelty):
The novelty of this work is somewhat limited, but the empirical findings are still worthwhile.

Author response:
We would like to highlight that the contributions of our framework come in two perspectives. 

First, we train an active exploration agent from intrinsic motivation to collect diverse and informative samples for both representation learning and downstream task models. This is different from previous works, where CRL [3] only actively pre-trains visual representations or SEAL [2] only fine-tunes a detection segmentation model using actively collected labeled samples. Our experiments demonstrate that ALP benefits from both actively learned visual representations (Table 2, 3) and actively collected task samples (Table 4), showing the importance of actively collecting samples for training vision models.

Second, given that the agent actively explores in visual environments, we propose a representation learning approach that *only* uses action information from embodied movements, which optimizes a reinforcement learning objective and an inverse dynamics objective. While prior works adopt similar ideas of “shared backbone” in supervised end-to-end RL tasks [5], we demonstrate that leveraging learning signals from actions can provide substantial gains in several vision tasks. This is different from popular visual representation learning methods on fixed or static datasets, such as contrastive loss [6, 7] or reconstruction error [8]. Importantly, we want to show that by actively exploring in environments the agent can effectively learn visual representations from embodied movements, as good as or better than baselines (Table 3). Our experiments demonstrate that ALP improves visual representation learning compared to self-supervised contrastive learning from the simulator (Table 2, 7) and each component contributes to better performance (Table 5).

We hope this could bring attention to the importance of actively collecting data for learning visual perception, both for pre-training visual representations and for fine-tuning task-specific models. This provides a different perspective from popular large-scale curated datasets, such as ImageNet and COCO, that require large amounts of human effort for data collection and annotations; instead our exploration agent is trained actively and autonomously from intrinsic motivation. Additionally, we *only* leverage action information in embodied settings as learning signals for visual representations, which is different from popular methods trained on fixed and static datasets. Through extensive experiments and ablations on a wide range of tasks we demonstrate benefits and gains from both perspectives.

Reviewer comment (Novelty):
The proposed training objective combining the policy gradient objective and the inverse dynamics model objective is similar to Ye et al. (2021). However, Ye et al. (2021) showed only for downstream navigation tasks.

Author response:
We would like to highlight that the contributions of our framework come in two perspectives. 

First, we train an active exploration agent from intrinsic motivation to collect diverse and informative samples for both representation learning and downstream task models. This is different from previous works, where CRL [3] only actively pre-trains visual representations or SEAL [2] only fine-tunes a detection segmentation model using actively collected labeled samples. Our experiments demonstrate that ALP benefits from both actively learned visual representations (Table 2, 3) and actively collected task samples (Table 4), showing the importance of actively collecting samples for training vision models.

Second, given that the agent actively explores in visual environments, we propose a representation learning approach that *only* uses action information from embodied movements, which optimizes a reinforcement learning objective and an inverse dynamics objective. While prior works adopt similar ideas of “shared backbone” in supervised end-to-end RL tasks [5], we demonstrate that leveraging learning signals from actions can provide substantial gains in several vision tasks. This is different from popular visual representation learning methods on fixed or static datasets, such as contrastive loss [6, 7] or reconstruction error [8]. Importantly, we want to show that by actively exploring in environments the agent can effectively learn visual representations from embodied movements, as good as or better than baselines (Table 3). Our experiments demonstrate that ALP improves visual representation learning compared to self-supervised contrastive learning from the simulator (Table 2, 7) and each component contributes to better performance (Table 5).

We hope this could bring attention to the importance of actively collecting data for learning visual perception, both for pre-training visual representations and for fine-tuning task-specific models. This provides a different perspective from popular large-scale curated datasets, such as ImageNet and COCO, that require large amounts of human effort for data collection and annotations; instead our exploration agent is trained actively and autonomously from intrinsic motivation. Additionally, we *only* leverage action information in embodied settings as learning signals for visual representations, which is different from popular methods trained on fixed and static datasets. Through extensive experiments and ablations on a wide range of tasks we demonstrate benefits and gains from both perspectives.

Reviewer comment (Theory):
Not clear why Inverse Dynamics Prediction is a good learning signal.

Author response:
**Inverse dynamics prediction as training objective**

While reinforcement learning objective provides a learning signal to associate visual frames to action, we additionally make an observation that consecutive frames encode agent’s movements in interactive settings. We design our inverse dynamics model to predict sequences of actions given observations, as shown in Figure 1. In this case, we directly infuse action information into visual representation learning, in addition to policy and reward information from reinforcement learning objectives. Our inverse dynamics model also provides temporal learning signals since it is trained on sequence of trajectories over multiple steps, different from RL policy that associate observation and action at every single step. Our ablation experiments in Table 5 clearly show that both of these components contribute to improved performance.

Reviewer comment (Theory):
What is the rationale behind IDM? It seems without reason to me, since the RL reward should already be a learning signal to associate visual frames to action.

Author response:
**Inverse dynamics prediction as training objective**

While reinforcement learning objective provides a learning signal to associate visual frames to action, we additionally make an observation that consecutive frames encode agent’s movements in interactive settings. We design our inverse dynamics model to predict sequences of actions given observations, as shown in Figure 1. In this case, we directly infuse action information into visual representation learning, in addition to policy and reward information from reinforcement learning objectives. Our inverse dynamics model also provides temporal learning signals since it is trained on sequence of trajectories over multiple steps, different from RL policy that associate observation and action at every single step. Our ablation experiments in Table 5 clearly show that both of these components contribute to improved performance.