From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by five experts in the field and received a mixed score. The main concerns are the limited novelty, unconvincing experiments, and lack of clarity. The authors did a good job of rebuttal and addressed many of the concerns. However, the reviewers and ACs still feel that more work is needed to get it to the best version. AC also agrees that this work can be much stronger with additional experiments and implementation details. While this paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.


justification_for_why_not_higher_score: 1. Limited novelty
2. The experiments are weak
3. The framework is not very general for real-world applications. 
justification_for_why_not_lower_score: NA

From: ICLR.cc/2023/Conference/Paper5396/Reviewer_hAS5
title: Post rebuttal
comment: Thanks for the authors' response. It addressed some of my concerns and clarified the unclear parts, I appreciate it.
While my main concerns about the technical novelty still remains, in this case I would like to keep my original rating, thanks!

From: ICLR.cc/2023/Conference/Paper5396/Reviewer_PQGj
title: Response
comment: Thank you for clarifying the problem setting, which is now clearer to me. I acknowledge the relevance of the new results added in the rebuttal. Adding comparisons with an adapted version of VPT in the future would surely be added value. My assessment of the paper remains overall positive.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Response Update
comment: Thank you for carefully reading our response. Please let us know if this addresses all your concerns, and if so, whether you will reconsider your overall score.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Response Update
comment: Thank you for carefully reading our response. Please let us know if this addresses all your concerns, and if so, whether you will reconsider your overall score.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Response Update
comment: Thank you for carefully reading our response. Please let us know if this addresses all your concerns, and if so, whether you will reconsider your overall score.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Response Update
comment: Thank you for carefully reading our response. Please let us know if this addresses all your concerns, and if so, whether you will reconsider your overall score.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Response Update
comment: Thank you for carefully reading our response. Please let us know if this addresses all your concerns, and if so, whether you will reconsider your overall score.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Response Part 3
comment: **It would be better to posit this paper as an offline RL paper and review literature related to learning from demonstrations, learning from observations, offline RL, and multi-task behavior cloning.**

We have added a offline RL section to the related work in the revision to give the work a more comprehensive positioning.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Initial Response Part 2
comment: **Other minor clarifications**

**Figure 1 reports that the trained IDM is "non-causal". The choice of a non-causal transformer IDM is not well discussed in the paper.**

We have added additional discussion around this (In the introduction).  The non-causal transformer is used so dynamics modeling information can be propagated in both directions.  It means the IDM can look at past and future frames in order to predict intermediate actions.  

In many experimental settings, the dynamics are far simpler than multi-faceted human behavior in the same setting. It has been suggested that IDMs are therefore more data efficient and this has also been empirically shown (Baker, 2022: https://arxiv.org/abs/2206.11795).

**What's "Dist(S)" reported in section 3.1?**

This refers to the simplex (set of distributions) over S. In Section 3.1, we state that our transition kernel of our MDP is a probability distribution over transitioning to a particular next state s’ conditioned on the current state and action.

**The second paragraph of Section 4.2 (During pretraining, ...") is a bit hard to follow because of different confusing statements. E.g., "we use the combined labelled dataset, i.e., D_d for all source environments M_d...", however, D_d is not the combined labelled dataset, but one of the datasets, right?. In contrast, the notation reported in Table 1 with the union of different D_d sets is more clear. I would suggest revising the whole section to improve clarity.**

A good point is made here and clarifications have been addressed in the updated manuscript. $D_d$ is a dataset for single source environments, of which there are $n$.  We have unified notation between Section 3.3, 4.2 and Table 1.

**The descriptions of the different baselines described in section 5.2 are hard to follow. I would suggest adding a summary table explaining the different properties of the baselines.**

We have added an additional table listing properties of the baselines in an easy to read checkmark notation. 

**Figure 2 reports "evaluation game performance". However, this evaluation measure is not mentioned or introduced in the main text.**

We have added an additional textual description to the Figure 2 caption of what this means. For Atari, this is the episodic return.

**The "Breakout" graph in Figure 4 does not have shaded lines (standard deviation).**

We have corrected this in the revision.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Initial Response
comment: Thanks for closely reading our submission! We address your feedback below.

**While I found the investigation overall interesting, one may object that the choice of tasks make the analysis and conclusions less general than what observed in other works.  Indeed, Baker et al (2022) considered the Minecraft game as its wide variety of things to do, build, and collect, can make results more applicable to varied real-world applications, while Lee et al. (2022) considered a set of 46 Atari games to show the abilities of their model to generalize across diverse scenarios. In contrast, the proposed investigation considers a subset of 5 Atari games.**

We have included new results, using an expanded set of up to 36 source games and 5 target games (Appendix Section A.2 and linked plots at: https://ibb.co/album/qkfs9Y ) to show a broader array of results. Performance is increased significantly by training on more source games.

**The main assumption is that a large quantity of labeled data is available for the source environments, while only a small set of labeled data in the target environment is available. Not surprisingly, the single-game variants, which are trained on the 10k subset of the data perform worse than methods trained on the set of 100M source + 10K target action labels. In order to assess the role of having more data as compared to being able to leverage data from diverse scenarios, one could consider the case in which single-game instances are trained on the target environments considering large sets of 100M labeled examples. This scenario is reasonable if we assume that we can label 100M examples on the target environment with the same effort required to label 100M examples on different source environments. More importantly, this setting would allow to assess the mere role of just using more data in the training procedure.**

We emphasize that our work studies a different setting, in which one already has access to a large set of related labelled training data. Thus, the choice is not about whether to spend effort to label the source datasets vs. the target dataset, but rather how to minimize the effort needed to label the target dataset. Our work shows that IDMs can be used to reduce the reliance on labelled target datasets. This is a much more attractive approach than the reviewer’s suggestion of manually labelling a 100M-sized dataset, especially in settings where one wishes to transfer to many downstream tasks. Rather than repeatedly collecting 100M labelled examples for each new task, we show that the knowledge from a labelled source dataset can be transferred to many tasks efficiently (see our newly added results in A.2 for a demonstration of this, finetuning a single pretrained model to multiple different target games).

**Another point which makes the experimental analysis a bit weak is the absence of clear comparisons with state-of-the-art approaches. For instance, one would expect comparisons with the VPT approach. If I understood correctly, this may be very similar to the "single-game" variants, but, if this is the case, this should be explicitly pointed out in the paper. Also, it could be interesting to see what happens if decision transformers are replaced by the methods considered for comparisons in Lee et al. (2022), such as C51, CQL, etc. These comparisons would help the reader to better understand the benefit of the proposed approach over current techniques.**

We appreciate the suggestion for additional baselines.  It is important to recognize the multi-game DT paper has made many of these comparisons (e.g., C51, CQL) and shown significant state of the art improvements over other methods.  Still, we agree that it may be useful to compare to VPT. Although the current opensource VPT codebase does not support discrete action spaces, we are currently examining ways to implement a good faith comparison to it. 

To make an additional comparison against a standard offline RL algorithm, we have included results for CQL in the Appendix Section A.4 with a dataset size of 10,000.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Initial Response Part 2
comment: 
**For figure 3, what are the selected source environments in “ALPT-2 Games” and “ALPT-10 Games”?**

We have added this clarification to the manuscript. ALPT-2 used ['Asterix', 'Breakout']. ALPT-10 used ['Asterix', 'Breakout', 'Freeway', 'Seaquest', 'SpaceInvaders', 'Atlantis', 'DemonAttack', 'Frostbite', 'Gopher', 'TimePilot'].

**The paper mentions that all the hyperparameters of DT are mostly the same as the settings in multi-game DT. It would be better to mention the key hyperparameters in the appendix for references, such as the number of heads and the number of layers.**

These parameter details have been added to the Appendix A.3 for clarification.

**What is the architecture of the IDM model?**

The IDM model is the same as the DT model, except that it is non-causal, which is enforced by changing the attention mask to an all-ones matrix.  We have added additional details to the Appendix A.3.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Initial Response Part 1
comment: Thanks for closely reading our submission! We address your feedback below.

**In section 4.2, the paper mentions that IDM and DT are trained simultaneously. However, it is still unclear what “simultaneously” means. Are IDM and DT trained iteratively? If so, what are the training steps for IDM (or DT) before switching to another model? What are the total training steps?**

IDM and DT are trained simultaneously, with a single optimizer. That is, at each training step we sample a batch of data for both the IDM and DT. We compute a loss for both and sum the losses before passing to an optimizer. There are a total of 1M pre-training steps, and we have clarified this in the updated manuscript.

**In finetuning, is DT initialized with the model pre-trained on all source environments? If so, it would be better to mention it clearly in section 4.2. Moreover, one ablation is initializing DT randomly and training it from scratch on the target datasets.**

Yes, it is initialized with the model trained on all source environments, along with the inferred labels from the IDM.  The suggested ablation is in Appendix Section A.1, using a pre-trained IDM and a randomly initialized DT.  We show that performance plateaus at a similar level in all games, suggesting that IDM pretraining is the main source of transfer benefits, as opposed to DT pretraining.

**In Figure 2, what is the meaning of the x-axis? Does it mean the number of labeled frames used in fine-tuning?**

This is the number of labeled game frames used during fine tuning. We have updated the caption to clarify this.

**In Figure 2, the evaluation curves of ALPT are quite bumpy across environments. What is the number of episodes used to get each evaluation point on the curve? Will a larger number of evaluation episodes help smooth the evaluation curve?**

We evaluate ALPT on 16 episodes of length 2500 each, following MGDT [1], which also showed bumpy curves (Figure 15 in the MGDT paper). A larger number of evaluation episodes should help smooth the curves, but we decided to use 16 following previous setups.  We have updated the caption of Figure 2 to clarify this.

[1] Lee, Kuang-Huei, et al. "Multi-Game Decision Transformers." arXiv preprint arXiv:2205.15241 (2022).

**Moreover, in Figure 2, ALPT does not seem to improve along finetuning, which indicates that the finetuning is not helpful. In Asterix, the performance even goes down after finetuning What may cause such experiment observations?** 

This may be an artifact of the way the figures are plotted and how the evaluation occurs.  Recall that we have at least 4000 warmup steps (See Appendix A.3 for experimental details) before any evaluation occurs.  As we know that finetuning is relatively quick, it is possible that all the finetuning is occurring within these 4000 steps, before the first evaluation.  This could explain why the first performance values that are plotted are already high.  Also, referring to the previous statement, bumpy curves were expected.

**In Figure 2, the DT-IDM baseline seems to have no shaded area. In Figure 4, the breakout experiments have no shaded area. Are they run with 3 random seeds?**

We have updated this figure to show the shaded area.

**Although ALPT improves upon baselines, the absolute rewards in Atari environments are still much lower than the reported values in the multi-game decision transformer’s appendix (https://arxiv.org/pdf/2205.15241.pdf page 24). For instance, DT’s performance in Asterix would have a reward of around 14,706 and ALPT has a reward of 2,500. It is reasonable that DT would perform better since it has the full dataset with actions. However, the large performance gap still raises concerns about the effectiveness of the proposed method.**

We would like to first highlight a few key differences.  The final scores of multigame DT, use all labeled actions, which is the primary reason for the performance gap.  Additionally, they use a larger model (DT-200M). We used 6 transformer layers (same as multigame DT-40M model) of 8 heads each.  As we can see in Figure 15 of Multigame DT, the model size has a significant impact on performance.  

Also the performance gap is not as wide for games other than Asterix.  In Seaquest for example, multigame DT achieves 5173 reward vs. ours which is slightly less than 3,500.  Another example is Freeway, where we in fact achieve higher reward.  In Breakout we achieve approximately 100-120 reward depending on fine tuning steps, whereas multigame DT achieves 290.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Initial Response
comment: Thanks for closely reading our submission! We address your feedback below.

**Although this paper provides an interesting perspective on large-scale training for RL, it's just a straightforward application of existing approaches, seems the novelty is not enough.**

We agree that our problem heavily leverages existing methods, and our focus is not to propose an entirely new algorithm. Rather, the focus of our contributions is (1) introduction of the ALPT setting, a practical yet under-explored problem setting, (2) results showing that IDM pretraining (despite its simplicity!) is a surprisingly performant and robust solution to this setting. We believe that discovering new capabilities for a standard approach on a difficult and highly practical problem is a worthy contribution to ICLR. 

**This paper only compared ALPT with the single-game variant and standard DT baselines, lacking enough experiments to support the proposed hypothesis.**

As we focus our experiments on the multi-game Atari setting, we compare to DT which is the state-of-the-art (see “Multi-Game Decision Transformers” by Lee, et al.). Please let us know if you have other experiments in mind. 

**Missing code**

We have added additional implementation and evaluation details in the Appendix Section A.3.  We will also release the anonymized code for our Maze task in the coming days.

We make the source code for our maze experiment available at:

https://anonymous.4open.science/r/alpt_maze-5927/README.md

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Initial Response
comment: Thanks for closely reading our submission! We address your feedback below.

**To achieve the goal, ALPT have to first pretrain two transformers, i.e. one bidirectional T for IDM and a DT, and these two Transformer need to be trained one by one (as training of DT will need the action labels generated by the bidirectional T). Then, ALPT needs an extra finetuning stage that have to the same thing, i.e. finetune two transformer one by one. I am wondering, all these process should be time and computation consuming, especially when dealing with massive scale datasets.**

We would like to clear up potential confusion about our method. We train the two transformers simultaneously with a single optimizer. We update the IDM and the DT parameters concurrently using the IDM loss and DT loss (as described in our manuscript) using a single optimizer.  Therefore, the increase in pretraining time is minimal; only additional parameters are required.  

We show evidence that this computational burden can be further decreased in Appendix A.1, where we show that pretraining the DT is often not necessary to achieve performance improvements.  Thus, if desired, pretraining can focus on the IDM exclusively.

**In experiments, only a couple of environments with small scale datasets are used for evaluation, which is hard to convince me that the proposed approach can indeed address lacking-of-label issue in large scale pretraining.**

We have included new results, using an expanded set of all source games (Appendix Section A.2 and linked plots at: https://ibb.co/album/qkfs9Y ) to show a broader array of results. These new experiments train on up to 36 source games (each of 100M transitions, totalling 3.6B) and 5 target games.  Performance is increased significantly by training on all games.

**The technical contribution and novelty is limited.**

We agree that our problem heavily leverages existing methods, and our focus is not to propose an entirely new algorithm. Rather, the focus of our contributions is (1) introduction of the ALPT setting, a practical yet under-explored problem setting, (2) results showing that IDM pretraining (despite its simplicity!) is a surprisingly performant and robust solution to this setting. We believe that discovering new capabilities for a standard approach on a difficult and highly practical problem is a worthy contribution to ICLR.

**Lack of implementation details and the evaluations are limited. No source codes are provided**

We have added additional implementation and evaluation details in the Appendix Section A.3.  We will also release the anonymized code for our Maze task in the coming days.

We make the source code for our maze experiment available at:

https://anonymous.4open.science/r/alpt_maze-5927/README.md

**When training IDM, it is saying that an bidirectional transformer is used, and is said that the model architecture as GPT. I am wondering GPT should be a causal transformer not bidirectional transformer, would you please clarify?** 

We have clarified in the work that our IDM transformer is non-causal, which is achieved by using the model architecture of GPT while modifying the attention mask (Appendix Section A.3).

**Is it interesting to see the different effects of training with IDM vs. forward dynamics model and even a combination of both forward and inverse dynamics model. I am wondering how and why that induce the choice of only IDM.**

We are confused by the term forward dynamics model, can the reviewer clarify?  IDM is a simple and natural approach to inference of actions.

**In sec. 5.5, only a pair of tasks are evaluated. When performing a single task with very simple action space, it very likely that the finetuning stage can already well fit the distribution. I am wondering a multitask setting or zero-shot/few-shot setting. At least without the IDM finetuning.** 

We have added to the paper an additional multi-task, multi-eval experiment (Appendix Section A.2 and linked plots at: https://ibb.co/album/qkfs9Y ). Please let us know if you have suggestions for additional experiments.

From: ICLR.cc/2023/Conference/Paper5396/Authors
title: Initial Response
comment: Thanks for closely reading our submission! We address your feedback below.

**The paper should clarify whether the performance difference comes mainly from using an IDM?**

We encourage the reviewer to see the Appendix A.1, where we compare the performance of our method after only pre-training the IDM, and then using the pre-trained IDM to infer actions, with a randomly initialized DT. These results show that performance improvements are still significant, with minimal additional training time required.  This suggests that pretraining the DT is not necessary and that performance improvements are tied mainly to the generalization capabilities of IDM, as argued in the intro section.

**The paper should also include a discussion on how the proposed method is able to leverage datasets with disjoint action spaces for improved performance.**

We hypothesize that performance improvements are not unexpected due to the already known broad generalization capabilities of DT architectures (Furuta 2022: https://openreview.net/pdf?id=CAjxVodl_v,  Reed 2022: https://openreview.net/forum?id=1ikK0kHjvj, Lee, 2022: https://arxiv.org/abs/2205.15241). In particular, Gato has shown generalization across discrete action atari games, continuous control, and language tasks. 

It may also be the case that, despite the action spaces being disjoint, they still exhibit similar structure. For example, a top/down action space vs. a left/right action space are structurally opposite to each other in terms of movement, differing only in orientation.  This similar structure is potentially learned and leveraged during finetuning. We have updated Section 5.5 to incorporate this discussion.

**The paper should provide a more detailed explanation and a discussion on why including more source games does not improve performance on the target environment (in Section 5.4).**

We point the reviewer to newly added results in Appendix A.2 Figure 7 (linked plots at: https://ibb.co/album/qkfs9Y ) using a much larger set of source and target environments. With these new results, we find that including more source games does show benefits. We will update Section 5.4 to note this for the final revision.

**The paper uses only five games as they have a similar shared game structure. This should be quantified in more detail. How similar does the game structure have to be? In general using only 5 games from the Atari suit is quite limited and the paper should ideally include more games.**

We have included new results (Appendix A.2, Fig 7) using an expanded set of up to 36 source games and 5 target evaluation games to show a broader array of results.  Please refer to the updated draft. 

**It would be interesting to include results on the performance difference when training purely on the fully label target environments as a percentage of recovered performance.**

We aim to include this in a final draft.

From: ICLR.cc/2023/Conference/Paper5396/Reviewer_NLz9
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper tackles the problem of limited availability of labelled datasets for reinforcement learning tasks. The paper proposes to circumvent this challenge by combining large but sparsely-annotated datasets from a target environment of interest (without densely annotated large scale datasets) with fully annotated datasets from various other source environments. The proposed method employs the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data. The method is evaluated on five games of the atArI benchmark suit.
strength_and_weaknesses: Strengths:
* The proposed method is interesting and novel. 
*  The proposed method seem to improve performance significantly.
* The paper includes adequate baselines and ablations.
* The paper is well written and easy to understand.


Weaknesses:
* Disjoint action spaces: the performance difference between ALPT and DT1-IDM is limited. The paper should clarify whether the performance difference comes mainly from using an IDM? The paper should also include a discussion on how the proposed method is able to leverage datasets with disjoint action spaces for improved performance.
* The paper should provide a more detailed explanation and a discussion on why including more source games does not improve performance on the target environment (in Section 5.4).
* The paper uses only five games as they have a similar shared game structure.  This should be quantified in more detail. How similar does the game structure have to be? In general using only 5 games from the Atari suit is quite limited and the paper should ideally include more games.
* It would be interesting to include results on the performance difference when training purely on the fully label target environments as a percentage of recovered performance. 
clarity,_quality,_novelty_and_reproducibility: Quality: The paper is interesting and novel.

Clarity: the paper is clear and well written.

Originality: To the best of my knowledge, the proposed idea is novel.
summary_of_the_review:  The claims in the paper are generally well founded with adequate baselines and ablations. The paper demonstrates impressive performance improvements. However, the paper should include more detailed dissuasions on some claims (see above).
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5396/Reviewer_hAS5
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studied the problem of learning agent polices with datasets that have limited labeled actions. Action Limited PreTraining (ALPT) is presented, which pretrain an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for decision transformer (DT) agent on an action limited target environment dataset. Experiments are conducted on game-playing benchmark among a couple of different environments. 
strength_and_weaknesses: Strength:
This work is well motivated. The research problem that's explored is valuable. Pretraining towards generic agents / representations on large-scale datasets has emerged as a dominant approach broadly in learning community. While, one big challenge exists in control and/or RL tasks is the lacking of explicitly labeled actions. Especially, when leveraging real-world video datasets, e.g. Ego4D for control tasks, the gap between semantic level 'activity' vs. physical level 'actions' brings ambiguous and large distribution discrepancies. So, to seek effective pretraining regimes that can leverage such massive datasets without actions labels is important. 

Weakness:
My main concern is that the proposed approach is hard to scale up. However, on contrary the main motivation of this work is for large-scale pretraining. To achieve the goal, ALPT have to first pretrain two transformers, i.e. one bidirectional T for IDM and a DT, and these two Transformer need to be trained one by one (as training of DT will need the action labels generated by the bidirectional T). Then, ALPT needs an extra finetuning stage that have to the same thing, i.e. finetune two transformer one by one. I am wondering, all these process should be time and computation consuming, especially when dealing with massive scale datasets. In experiments, only a couple of environments with small scale datasets are used for evaluation, which is hard to convince me that the proposed approach can indeed address lacking-of-label issue in large scale pretraining. 
clarity,_quality,_novelty_and_reproducibility: 1. The technical contribution and novelty is limited. As what I mentioned before, the motivation of this work is to address issues for large-scale pretraining. However, both the techniques and experiments do not well validate the claim. The key novelty of the proposed approach is to leverage source and part of labeled target environments to train a model that can generate action labels. It closely related with widely used ideas of generating pseudo labels in transfer learning and semi-supervised learning.
2. Lack of implementation details and the evaluations are limited. No source codes are provided.
Some questions:
1) When training IDM, it is saying that an bidirectional transformer is used, and is said that the model architecture as GPT. I am wondering GPT should be a causal transformer not bidirectional transformer, would you please clarify?
2) it it interesting to see the different effects of training with IDM vs. forward dynamics model and even a combination of both forward and inverse dynamics model. I am wondering how and why that induce the choice of only IDM.
3) In sec. 5.5, only a pair of tasks are evaluated. When performing a single task with very simple action space, it very likely that the finetuning stage can already well fit the distribution. I am wondering a multitask setting or zero-shot/few-shot setting. At least without the IDM finetuning. 
summary_of_the_review: This paper studied the problem of learning agent polices with datasets that have limited labeled actions. Action Limited PreTraining (ALPT) is presented, which pretrain an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for decision transformer (DT) agent on an action limited target environment dataset. Experiments are conducted on game-playing benchmark among a couple of different environments. 
This work is well motivated. The research problem that's explored is valuable. Pretraining towards generic agents / representations on large-scale datasets has emerged as a dominant approach broadly in learning community. While, one big challenge exists in control and/or RL tasks is the lacking of explicitly labeled actions. Especially, when leveraging real-world video datasets, e.g. Ego4D for control tasks, the gap between semantic level 'activity' vs. physical level 'actions' brings ambiguous and large distribution discrepancies. So, to seek effective pretraining regimes that can leverage such massive datasets without actions labels is important. 
However, my main concern is that the proposed approach is hard to scale up. However, on contrary the main motivation of this work is for large-scale pretraining. To achieve the goal, ALPT have to first pretrain two transformers, i.e. one bidirectional T for IDM and a DT, and these two Transformer need to be trained one by one (as training of DT will need the action labels generated by the bidirectional T). Then, ALPT needs an extra finetuning stage that have to the same thing, i.e. finetune two transformer one by one. I am wondering, all these process should be time and computation consuming, especially when dealing with massive scale datasets. In experiments, only a couple of environments with small scale datasets are used for evaluation, which is hard to convince me that the proposed approach can indeed address lacking-of-label issue in large scale pretraining. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5396/Reviewer_aonF
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper explores the problem of learning agent policies from action limited datasets and proposes ALPT, which pretrains an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for a decision transformer (DT) agent on an action limited target environment dataset. The experiments and ablations highlight the importance of pretraining the IDM, as opposed to pretraining the DT agent alone. The results support the importance of generalist inverse dynamics models as an efficient way to implement large-scale RL. In addition, as more labelled data becomes available for training offline RL agents, ALPT provides an efficient way of bootstrapping performance on new tasks with action limited data.
strength_and_weaknesses: ##########################################################################

Pros:

- The paper proposes ALPT to use the multi-environment source datasets as pretraining for an IDM, which is then fine-tuned on the action-limited data of the target environment in order to provide labels for the unlabelled target data, which is then used for training a DT agent, which is an interesting idea for learning agent policies from action limited datasets.
- Through various experiments and ablations, this paper demonstrates that leveraging the generalization capabilities of IDMs is critical to the success of ALPT, as opposed to, for example, pretraining the DT model alone on the multi-environment datasets or training the IDM only on the target environment. In addition, on a benchmark game-playing environment, we show that ALPT yields as much as 5x improvement in performance, with as little as 10k labelled samples required (i.e., 0.01% of the original labels), derived from only 12 minutes of labelled game play (Ye et al., 2021).
- The paper is well-written and the comparison of benchmark methods is also interesting to read.
##########################################################################

Cons:

- Although this paper provides an interesting perspective on large-scale training for RL, it's just a straightforward application of existing approaches, seems the novelty is not enough.
- This paper only compared ALPT with the single-game variant and standard DT baselines, lacking enough experiments to support the proposed hypothesis.
- Missing code link to reproduce experiments.
clarity,_quality,_novelty_and_reproducibility: The quality and clarity are good but the originality is only marginally significant or novel, and missed code links to reproduce experiments.
summary_of_the_review: Considering the above pros and cons, my recommendation of the paper is marginally below the acceptance threshold.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5396/Reviewer_qasR
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper is motivated by the limited availability of actions in datasets when training RL agents. To better leverage the large amount of data not paired with actions, this paper proposes to learn a model to annotate actions for the dataset. More concretely, this paper proposes Action Limited PreTraining (ALPT), which learns an inverse dynamics model (IDM) to label missing actions for datasets in the target environment. ALPT is evaluated in Atari environments and maze navigation environments. ALPT has two stages: the pre-training stage, which trains a DT model and IDM simultaneously on datasets collected in source environments, and the finetuning stage, which finetunes both models on datasets in the target environment. 
strength_and_weaknesses: Strength:
 - The paper is well-motivated and focuses on realistic problems. The action-limited problem focused in this paper stems from the widely studied learning-from-observation problem. 
 - The paper is well-organized and easy to follow.
 - The motivation is clearly stated. 

Weakness:

 - Unclear methodology description.
     - In section 4.2, the paper mentions that IDM and DT are trained simultaneously. However, it is still unclear what “simultaneously” means. Are IDM and DT trained iteratively? If so, what are the training steps for IDM (or DT) before switching to another model? What are the total training steps?
     - In finetuning, is DT initialized with the model pre-trained on all source environments? If so, it would be better to mention it clearly in section 4.2. Moreover, one ablation is initializing DT randomly and training it from scratch on the target datasets. 
 
- Concerns regarding the Atari experiments
     - In Figure 2, what is the meaning of the x-axis? Does it mean the number of labeled frames used in fine-tuning? 
     - In Figure 2, the evaluation curves of ALPT are quite bumpy across environments. What is the number of episodes used to get each evaluation point on the curve? Will a larger number of evaluation episodes help smooth the evaluation curve?
     -  **Moreover, in Figure 2, ALPT does not seem to improve along finetuning, which indicates that the finetuning is not helpful. In Asterix, the performance even goes down after finetuning What may cause such experiment observations? One ablation to do is to freeze the IDM model and only finetune DT.
     - In Figure 2, the DT-IDM baseline seems to have no shaded area. In Figure 4, the breakout experiments have no shaded area. Are they run with 3 random seeds?
     - Although ALPT improves upon baselines, the absolute rewards in Atari environments are still much lower than the reported values in the multi-game decision transformer’s appendix (https://arxiv.org/pdf/2205.15241.pdf page 24). For instance, DT’s performance in Asterix would have a reward of around 14,706 and ALPT has a reward of 2,500. It is reasonable that DT would perform better since it has the full dataset with actions. However, the large performance gap still raises concerns about the effectiveness of the proposed method.

- Lacking experimental setup descriptions.
     - For figure 3, what are the selected source environments in “ALPT-2 Games” and “ALPT-10 Games”?  
     - The paper mentions that all the hyperparameters of DT are mostly the same as the settings in multi-game DT. It would be better to mention the key hyperparameters in the appendix for references, such as the number of heads and the number of layers. 
     - What is the architecture of the IDM model?  
- Missing key related works in the literature review. 
     - It would be better to posit this paper as an offline RL paper and review literature related to learning from demonstrations, learning from observations, offline RL, and multi-task behavior cloning. 
clarity,_quality,_novelty_and_reproducibility: The motivation of the paper is clear. The quality of the literature review, methodology, and experiments can still be largely improved. 
summary_of_the_review: I would recommend a major revision of the paper and would recommend rejecting the paper. Although the paper is well-motivated, there are key descriptions missing in the methodology part, which makes the paper hard to evaluate. Moreover, the experiments hardly show that finetuning indeed helps.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5396/Reviewer_PQGj
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes an approach to train offline reinforcement learning methods with a mix of data belonging to the target environment or task of interest (e.g., a game) and data belonging to different, but related environments/tasks (e.g., different games). In particular, the paper proposes to use a small set of labeled data from the target environment, plus a large set of unlabeled data from the target environment and a large set of labeled data from the source environments. By using an approach similar to Vide PreTraining, an Inverse Dymanics Model is trained on labeled data and used to annotate unlabeled data. All the data is used to train the final policy. This approach results in the proposed ALPT method. Experiments compare ALPT with a series of baselines to show the appropriateness of the different steps of the proposed approach. Other experiments aim to give insights into the role of dynamics modeling, the effect of using source and target environments with disjoint action spaces, and the benefit of ALPT in other domains such as maze navigation.
strength_and_weaknesses: STRENGHTS:
 - While the proposed technique is conceptually simple and the technical novelty with respect to previous works such as VPT (Baker et al., 2022), the proposed investigation seems to be useful to me. Indeed, it is reasonable to assume that labeled examples from similar tasks will become are available or will become available with time. In this case, it can be useful to be able to leverage such annotations to improve performance on a related task. The experimental analysis is also carried out trying to answer interesting questions concerning the role of each step of the proposed training pipeline, the effect of the number of source datasets used for dynamics modeling, the performance of the ALPT in the presence of disjoint action spaces, and the benefits of the proposed technique in other domains.

WEAKNESSES:
 - While I found the investigation overall interesting, one may object that the choice of tasks make the analysis and conclusions less general than what observed in other works. Indeed, Baker et al (2022) considered the Minecraft game as its wide variety of things to do, build, and collect, can make results more applicable to varied real-world applications, while Lee et al. (2022) considered a set of 46 Atari games to show the abilities of their model to generalize across diverse scenarios. In contrast, the proposed investigation considers a subset of 5 Atari games (asterix, breakout, freeway, seaquest, and spaceinvaders) which, while maintaining a certain degree of variety, are arguably easier to tackle than Minecraft or the full set of 46 Atari games. In my opinion, this choice makes the paper and results somewhat weaker as compared to the two aforementioned works.
 - The paper reports reasonable ablations and comparisons. However, the main assumption is that a large quantity of labeled data is available for the source environments, while only a small set of labeled data in the target environment is available. Not surprisingly, the single-game variants, which are trained on the 10k subset of the data perform worse than methods trained on the set of 100M source + 10K target action labels. In order to assess the role of having more data as compared to being able to leverage data from diverse scenarios, one could consider the case in which single-game instances are trained on the target environments considering large sets of 100M labeled examples. This scenario is reasonable if we assume that we can label 100M examples on the target environment with the same effort required to label 100M examples on different source environments. More importantly, this setting would allow to assess the mere role of just using more data in the training procedure.
 - Another point which makes the experimental analysis a bit weak is the absence of clear comparisons with state-of-the-art approaches. For instance, one would expect comparisons with the VPT approach. If I understood correctly, this may be very similar to the "single-game" variants, but, if this is the case, this should be explicitly pointed out in the paper. Also, it could be interesting to see what happens if decision transformers are replaced by the methods considered for comparisons in Lee et al. (2022), such as C51, CQL, etc. These comparisons would help the reader to better understand the benefit of the proposed approach over current techniques.
clarity,_quality,_novelty_and_reproducibility: I found the paper not always very clear and easy to follow. For instance: 
 - Figure 1 reports that the trained IDM is "non-causal". The choice of a non-causal transformer IDM is not well discussed in the paper.
 - What's "Dist(S)" reported in section 3.1?
 - The second paragraph of Section 4.2 (During pretraining, ...") is a bit hard to follow because of different confusing statements. E.g., "we use the combined labelled dataset, i.e., D_d for all source environments M_d...", however, D_d is not the combined labelled dataset, but one of the datasets, right?. In contrast, the notation reported in Table 1 with the union of different D_d sets is more clear. I would suggest revising the whole section to improve clarity.
 -  The descriptions of the different baselines described in section 5.2 are hard to follow. I would suggest adding a summary table explaining the different properties of the baselines.
 - Figure 2 reports "evaluation game performance". However, this evaluation measure is not mentioned or introduced in the main text.
 - The "Breakout" graph in Figure 4 does not have shaded lines (standard deviation).

The technical novelty of the paper is not very high. Indeed, as the authors state, ALPT is a straightforward application of existing approaches. Nevertheless, I found the conceptual novelty of the paper good.
summary_of_the_review: In summary, while I found the investigation and analysis very interesting, in my opinion, the quality of the paper is limited by the aspects outlined above. I believe a careful revision of those aspects would make the submission much more solid.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold