Original: Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
Revised: Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

Revised: Anonymous Author(s)

Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: Pretraining on noisy, internet-scale datasets has been heavily studied as a techniquefor training models with broad, general capabilities for text, images, and othermodalities.
Revised: Pretraining on noisy, internet-scale datasets has been heavily studied as a techniquefor training models with broad, general capabilities for text, images, and othermodalities.

Original: However, for many sequential decision domains such as robotics,video games, and computer use, publicly available data does not contain the labelsrequired to train behavioral priors in the same way.
Revised: However, for many sequential decision domains such as robotics,video games, and computer use, publicly available data does not contain the labelsrequired to train behavioral priors in the same way.

Original: We extend the internet-scalepretraining paradigm to sequential decision domains through semi-supervisedimitation learning wherein agents learn to act by watching online unlabeled videos.
Revised: We extend the internet-scalepretraining paradigm to sequential decision domains through semi-supervisedimitation learning wherein agents learn to act by watching online unlabeled videos.

Original: Speciﬁcally, we show that with a small amount of labeled data we can train aninverse dynamics model accurate enough to label a huge unlabeled source of onlinedata – here, online videos of people playing Minecraft – from which we can thentrain a general behavioral prior.
Revised: Speciﬁcally, we show that with a small amount of labeled data we can train aninverse dynamics model accurate enough to label a huge unlabeled source of onlinedata – here, online videos of people playing Minecraft – from which we can thentrain a general behavioral prior.

Original: Despite using the native human interface (mouseand keyboard at 20Hz), we show that this behavioral prior has nontrivial zeroshot capabilities and that it can be ﬁne-tuned, with both imitation learning andreinforcement learning, to hard-exploration tasks that are impossible to learn fromscratch via reinforcement learning.
Revised: Despite using the native human interface (mouseand keyboard at 20Hz), we show that this behavioral prior has nontrivial zeroshot capabilities and that it can be ﬁne-tuned, with both imitation learning andreinforcement learning, to hard-exploration tasks that are impossible to learn fromscratch via reinforcement learning.

Original: For many tasks our models exhibit humanlevel performance, and we are the ﬁrst to report computer agents that can craftdiamond tools, which can take proﬁcient humans upwards of 20 minutes (24,000environment actions) of gameplay to accomplish.
Revised: For many tasks our models exhibit humanlevel performance, and we are the ﬁrst to report computer agents that can craftdiamond tools, which can take proﬁcient humans upwards of 20 minutes (24,000environment actions) of gameplay to accomplish.

Original:  Introduction
Revised:  Introduction

Original: Work in recent years has demonstrated the efﬁcacy of pretraining large and general foundationmodels 7 on noisy internet-scale datasets for use in downstream tasks in natural language 1–4 , computervision, 5,6,8 and multi-task models.
Revised: Work in recent years has demonstrated the efﬁcacy of pretraining large and general foundationmodels 7 on noisy internet-scale datasets for use in downstream tasks in natural language 1–4 andcomputer vision.

Original: 9 For sequential decision domains (e g. robotics, game playing,and computer usage) where agents must repeatedly act within an environment, a wealth of data alsoexists on the web; however, most of this data is in the form of unlabeled video (i e. without theactions taken at each frame), making it much less straightforward to train a behavioral prior in thesedomains than it is in e.g. natural language.
Revised: For sequential decision domains (e g. robotics, game playing and computerusage) where agents must repeatedly act within an environment, a wealth of data also exists on theweb; however, most of this data is in the form of unlabeled video (i e. without the actions takenat each frame), making it much less straightforward to train a behavioral prior in these domainsthan it is in e.g. natural language.

Original: In a few rare settings, such as Chess, Go, and StarCraft,there already exist large datasets with action labels from various online platforms that researchershave used for imitation learning.
Revised: In a few rare settings, such as Chess, Go, and StarCraft, therealready exist large datasets with action labels from various online platforms that researchers haveused for imitation learning.

Original: 10,11 When large labeled datasets do not exist, the canonical strategyfor training capable agents is reinforcement learning (RL), 12 which can be sample inefﬁcient andexpensive for hard-exploration problems.
Revised: When large labeled datasets do not exist, the canonical strategyfor training capable agents is reinforcement learning (RL), 11 which can be sample inefﬁcient andexpensive for hard-exploration problems.

Original:  Many virtual tasks, e.g. navigating websites, using
Revised: 12–18 Many virtual tasks, e.g. navigating websites, using

Original: Photoshop, booking ﬂights, etc., can be very hard to learn with RL and do not have large, commonlyavailable sources of labeled data.
Revised: Photoshop, booking ﬂights, etc., can be very hard to learn with RL and do not have large, commonlyavailable sources of labeled data.

Original: In this paper, we seek to extend the paradigm of traininglarge, general-purpose foundation models to sequential decision domains by utilizing freely availableinternet-scale unlabeled video datasets with a simple semi-supervised imitation learning method.
Revised: In this paper, we seek to extend the paradigm of traininglarge, general-purpose foundation models to sequential decision domains by utilizing freely availableinternet-scale unlabeled video datasets with a simple semi-supervised imitation learning method.

Original: Wecall this method Video PreTraining (VPT) and demonstrate its efﬁcacy in the domain of Minecraft.
Revised: Wecall this method video pretraining (VPT).

Original: Existing semi-supervised imitation learning methods aim to learn with few or no explicit action labels;however, they generally rely on the policy’s ability to explore the environment throughout training,making them susceptible to exploration bottlenecks.
Revised: Existing semi-supervised imitation learning methods aim to learn with few or no explicit actionlabels. However, they can be complex and difﬁcult to tune, e.g. balancing adversarial objectives 23or relying on policy rollouts to construct world models.

Original: 22–26 Furthermore, most prior semi-supervisedimitation learning work was tested in the relatively low data regime; because we experiment with farmore data ( ⇠ 70k hours of unlabeled video), we hypothesize that we can achieve good performancewith a much simpler method, a trend that has proven true for pretraining in other modalities such as text.
Revised: 24,26 Furthermore, most prior semi-supervisedimitation learning work was tested in the relatively low data regime; because we experiment with farmore data ( ⇠ 70k hours of unlabeled video), we hypothesize that we can achieve good performancewith a much simpler method, a trend that has proven true for pretraining in other modalities suchas text.

Original: 1 In particular, given a large but unlabeled dataset, we propose generating pseudo-labels bygathering a small amount of labeled data to train an inverse dynamics model (IDM) that predictsthe action taken at each timestep in a video.
Revised: 1 In particular, given a large but unlabeled dataset, we propose generating pseudo-labels bygathering a small amount of labeled data to train an inverse dynamics model (IDM) that predictsthe action taken at each timestep in a video.

Original: Behavioral cloning (BC) can require a large amountof data because the model must learn to infer intent and the distribution over future behaviors fromonly past observations.
Revised: Behavioral cloning (BC) can require a large amountof data because the model must learn to infer intent and the distribution over future behaviors fromonly past observations.

Original: In contrast, the inverse dynamics modeling task is simpler because it isnon-causal , meaning it can look at both past and future frames to infer actions.
Revised: In contrast, the inverse dynamics modeling task is simpler because it isnon-causal , meaning it can look at both past and future frames to infer actions.

Original: In most settings,environment mechanics are far simpler than the breadth of human behavior that can take place withinthe environment, suggesting that non-causal IDMs could require far less data to train than causal BCmodels.
Revised: In most settings,environment mechanics are far simpler than the breadth of human behavior that can take place withinthe environment, suggesting that non-causal IDMs could require far less data to train than causal BCmodels.

Original: Using pseudo-labels generated from the IDM, we then train a model to mimic the distributionof behavior in the previously unlabeled dataset with standard behavioral cloning at scale, which doesnot require any model rollouts and thus does not suffer from any potential exploration bottlenecksin the environment.
Revised: Using pseudo-labels generated from the IDM, we then train a model to mimic the distributionof behavior in the previously unlabeled dataset with standard behavioral cloning at scale.

Original: Finally, we show we can ﬁne-tune this model to downstream tasks with eitherbehavioral cloning or reinforcement learning.
Revised: Finally, weshow we can ﬁne-tune this model to downstream tasks with either behavioral cloning or RL.

Original: We chose to test our method in  Minecraft  becauseit (a)  is oneof the most actively played games in the world 27 and thus has awealth of online video data, (b)  is an open-ended sandbox gamewith an extremely wide variety of potential things to do, build,and collect, making our results more applicable to real-worldapplications such as computer usage, which also tends to bevaried and open-ended, and (c)  has already garnered interest bythe RL community as a research domain due to its complexity andcorrespondingly difﬁcult exploration challenges.
Revised: We chose to test our method in Minecraft. Minecraft is an idealtesting ground because (a) it is one of the most actively playedgames in the world 27 and thus has a wealth of commonly availablevideo data online, (b) it is a fairly open-ended sandbox gamewith an extremely wide variety of potential things to do, build,and collect, making our results more applicable to real-worldapplications such as computer usage, which also tends to bevaried and open-ended, and (c) it has already garnered interestby the RL community as a research domain due to its complexityand correspondingly difﬁcult exploration challenges.

Original: In this workwe use the native human interface for Minecraft so that we can(1) most accurately model the human behavior distribution andreduce domain shift between video data and the environment, (2)make data collection easier by allowing our human contractors to play the game without modiﬁcation,and (3) eliminate the need to hand-engineer a custom interface for models to interact with theenvironment.
Revised: In thiswork we use the native human interface for Minecraft so that wecan (1) most accurately model the human behavior distributionand reduce domain shift between video data and the environment, (2) make data collection easier, asour contractors can play the game without modiﬁcation, and (3) eliminate the need to hand-engineer acustom interface for models to interact with the environment.

Original: This choice means that our models play at 20 frames per second and must use amouse and keyboard interface to interact with human GUIs for crafting, smelting, trading, etc. ,including dragging items to speciﬁc slots or navigating the recipe book with the mouse cursor (Fig. 1).
Revised: This choice means that our models playat 20 frames per second and must use a mouse and keyboard interface to interact with human GUIsfor crafting, smelting, trading, etc., including dragging items to speciﬁc slots or navigating the recipebook with the mouse cursor (Fig. 1).

Original: Compared to prior work in Minecraft that uses a lower frame rate and constructs crafting and attackingmacros, 31,33–35 using the native human interface drastically increases the environment’s explorationdifﬁculty, making most simple tasks near impossible with RL from scratch.
Revised: Compared to prior work in Minecraft that uses a lower framerate and constructs crafting and attacking macros, 31,33–35 using the native human interface drasticallyincreases the environment’s exploration difﬁculty, making most simple tasks near impossible with RL from scratch.

Original: Even the simple task ofgathering a single wooden log while already facing a tree takes 60 consecutive attack actions with thehuman interface, meaning the chance for a naive random policy to succeed is 1 / 2 60 .
Revised: Even the simple task of gathering a single wooden log while already facing a treetakes 60 consecutive attack actions with the human interface, meaning the chance for a naive randompolicy to succeed is 1260 .

Original: While this papershows results in Minecraft only, the VPT method is general and could be applied to any domain.
Revised: While this paper shows results in Minecraft only, the VPT method is generaland could be applied to any domain.

Original: In Section 4 we show that the VPT foundation model has nontrivial zero-shot performance, accomplishing tasks impossible to learn with RL alone, such as crafting planks and crafting tables (tasksrequiring a human proﬁcient in Minecraft a median of 50 seconds or ⇠ 970 consecutive actions).
Revised: In Section 4 we show that the VPT foundation model has nontrivial zero-shot performance, accomplishing tasks impossible to learn with RL alone, such as crafting planks and crafting tables (tasksrequiring a human proﬁcient in Minecraft a median of 50 seconds or ⇠ 970 consecutive actions).

Original: Through ﬁne-tuning with behavioral cloning to smaller datasets that target more speciﬁc behaviordistributions, our agent is able to push even further into the technology tree, crafting stone tools(taking a human a median of 2.3 minutes or ⇠ 2790 actions).
Revised: Through ﬁne-tuning with behavioral cloning to smaller datasets that target more speciﬁc behaviordistributions, our agent is able to push even further into the technology tree, crafting stone tools(taking a human a median of 2.3 minutes or ⇠ 2790 actions).

Original: Finally, ﬁne-tuning via RL producesthe most dramatic improvements: our agent is able to craft diamond tools, an unprecedented resultin Minecraft made even more challenging by using the native human interface.
Revised: Finally, ﬁne-tuning via RL produces the most dramatic improvements: our agent is able to craft diamond tools, an unprecedented result in Minecraft made even more challenging by using the native human interface.

Original: This task requiresa proﬁcient human a median upwards of 20 minutes or ⇠ 24000 actions.
Revised: This task requires a proﬁcient human a median upwards of 20 minutes or ⇠ 24000 actions.

Original: The main contributionsof this work are (1) we are the ﬁrst to show promising results applying semi-supervised imitationlearning to extremely large, noisy, and freely available video datasets for sequential decision domains,(2)
Revised: The main contributions of this work are (1) we are the ﬁrst to show promising results applying semi-supervised imitation learning to extremely large, noisy, and freely available video datasets for sequential decision domains

Original:  we show that such pretraining plus ﬁne-tuning enables agents to solve tasks that were otherwiseimpossible to learn, (3) we show that labeled contractor data is far more efﬁciently used withinthe VPT method than it would be by directly training a foundation model from it and (4) we opensource our contractor data, trained model weights, and Minecraft environment for future researchinto learning to act via semi-supervised imitation learning at scale.
Revised: (2) we show that such pretraining plus ﬁne-tuning enables agents to solve tasks that were otherwise impossible to learn, (3) we show that labeled contractor data is far more efﬁciently used within the VPT method than it would be by directly training a foundation model from it and (4) we open source our contractor data, trained model weights and Minecraft environment for future research into learning to act via semi-supervised imitation learning at scale Train the VPT Foundation Model via Behavioral Cloning [Figure] keywords

Revised: [Figure]

Original:  Preliminaries and Related Work
Revised:  Preliminaries and Related Work

Original: Imitation learning methods 36–39 seek to construct a policy that accurately models the distribution ofbehavior in some dataset D
Revised: Imitation learning methods 36–39 seek to construct a policy that accurately models the distribution ofbehavior in some dataset D

Original: = { ( o i , a i ) } , i 2 { 1 ...N } of action-observation pairs.
Revised: = { ( o i , a i ) } , i 2 { 1 . . . N } of action-observation pairs.

Original: In order to roll 96 out these policies in an environment, they must be causal , meaning they condition on observationsfrom the current timestep t and past timesteps only, i.e. ⇡ ⇠ p ( a t | o 1 ...o t ) .Imitation learning is Training the VPT Foundation Model via Behavioral Cloning [Figure]
Revised: In order to roll 95 out these policies in an environment, they must be causal , meaning they condition on observationsfrom the current timestep t and past timesteps only, i.e. ⇡ ⇠ p ( a t | o 1 ...o t ) .

Original: simplest when demonstrations are labeled with corresponding actions.Imitating labeled trajectories
Revised: Imitation learning issimplest when demonstrations are labeled with corresponding actions.

Original: [Figure] has seen success in aerial vehicles, 40,41 self-driving cars, 42,43 board games, 10,44 and video games.[Figure]
Revised: Imitating labeled trajectorieshas seen success in aerial vehicles, 40,41 self-driving cars, 42,43 board games, 9,44 and video games.

Original: When labeled demonstrations are not available, standard behavioral cloning will not work; however, [Figure] there is a large body of work in imitating behavior from unlabeled demonstrations.
Revised: When labeled demonstrations are not available, standard behavioral cloning will not work; however,there is a large body of work in imitating behavior from unlabeled demonstrations.

Original: For instance,
Revised: For instance,

Original: [Figure] GAIL 24 constructs an adversarial objective incentivizing the trained policy to exhibit behaviors [Figure] indistinguishable from those in the target dataset.
Revised: GAIL 23 constructs an adversarial objective incentivizing the trained policy to exhibit behaviorsindistinguishable from those in the target dataset.

Original: Edwards et al.
Revised: Edwards et al.

Original: 46 propose to ﬁrst learn a latent [Figure] policy using unlabeled demonstrations and then map the learned latent actions to real actions usingenvironment interaction.
Revised: 26 propose to ﬁrst learn a latentpolicy using unlabeled demonstrations and then map the learned latent actions to real actions witha small amount of environment interaction.

Original: Peng et al.
Revised: Peng et al.

Original: 47 use motion-capture methods to track agent positions in videosand then train RL agents to match these waypoints.
Revised: 46 ﬁrst use motion-capture methods to trackagent positions in videos and then train RL agents to match these waypoints.

Original: Similarly, Behbahani et al.
Revised: Similarly, Behbahaniet al.

Original: 48 and Aytar et al.
Revised: 47 and Aytar et al.

Original: [Figure] task a RL agent to match waypoints; however, their waypoints are embeddings from unsupervisedfeature learning models.
Revised: 48 task a RL agent to match waypoints; however, they construct waypoints thatare embeddings from unsupervised feature learning models.

Original: Pathak et al.
Revised: Pathak et al.

Original: 50 and Nair et al.
Revised: 49 and Nair et al.

Original: 51 train goal conditioned policies to takeactions that move towards expert-provided goal states expressed as high dimensional visual waypoints.
Revised: 50 traingoal conditioned policies to take actions that advance the current state towards expert-provided goalstates expressed as high dimensional visual waypoints.

Original: Most similar to our own work, Torabi et al.
Revised: Most similar to our own work, Torabi et al.

Original: 25 simultaneously train (1) an inverse dynamics model(IDM), 52 which aims to uncover the underlying action between timesteps given observations of pastand future timesteps, e.g. p IDM ( a t | o t , o t +1 ) , and (2) a behavioral cloning (BC) model on trajectoriesof observations labeled with the IDM.
Revised: 24simultaneously train (1) an inverse dynamics model (IDM), 51 which aims to uncover the underlyingaction between timesteps given observations of past and future timesteps, e.g. p IDM ( a t | o t , o t +1 ) ,and (2) a behavioral cloning (BC) model on trajectories of observations labeled with the IDM.

Original: Data to train the IDM is collected by rolling out the BCmodel in the target environment such that both models improve in tandem.
Revised: Datato train the IDM is collected by rolling out the BC model in the target environment such that bothmodels improve in tandem.

Original: However, at any point intraining if there are sequences in the dataset that the IDM performs poorly on, it requires that the BCmodel perform those or similar sequences in order for the IDM to improve and correctly label them.
Revised: However, at any point in training if there are sequences in the dataset thatthe IDM performs poorly on, it requires that the BC model perform those or similar sequences inorder for the IDM to improve and correctly label them.

Original: Therefore, if the BC model does not explore efﬁciently, it could severely slow down learning.
Revised: Therefore, if the BC model does not exploreefﬁciently, it could severely slow down learning.

Original: Inorder to avoid this potential issue we opted for a simpler two-stage approach: we ﬁrst train an IDMon a small number of labeled trajectories collected from human contractors (they play the game aswould normally as we record their keypresses and mouse movements).
Revised: In order to avoid this potential issue we opted for asimpler two-stage approach: we ﬁrst train an IDM on a small number of labeled trajectories collectedfrom human contractors.

Original: Because human contractorsreach most relevant parts of the state space, we can hold the IDM ﬁxed throughout BC training.
Revised: Because human contractors reach most relevant parts of the state space, wecan hold the IDM ﬁxed throughout BC training.

Original: Compared to most previous work in semi-supervised imitation learning, we experiment in the muchmore complex and open-ended environment of Minecraft.
Revised: Compared to most previous work in semi-supervised imitation learning, we experiment in the muchmore complex and open-ended environment of Minecraft.

Original: Minecraft is a voxel-based 3D videogame that, due its popularity and wide variety of mechanics, has attracted a vast amount of RLresearch.
Revised: Minecraft is a voxel-based 3D videogame that, due its popularity and wide variety of mechanics, has attracted a vast amount of RLresearch.

Original: A large body of work focuses on small, custom-made Minecraft worldswith tasks such as navigation, 54,61 block placing, 55,56 instruction following, 59,60 combat, 57 andothers.
Revised: A large body of work focuses on small, custom-made Minecraft worldswith tasks such as navigation, 53,60 block placing, 54,55 instruction following, 58,59 combat, 56 andothers.

Original: 29,32,58 Work operating in the massive, randomly generated environments of Minecraft itselfhas included hill climbing, 53 automated curriculum learning 31 and, most closely related to the RLexperiments presented in Sec. 4.4, diamond mining.
Revised: 29,32,57 Work operating in the massive, randomly generated environments of Minecraft itselfhas included hill climbing, 52 automated curriculum learning 31 and, most closely related to the RLexperiments presented in Sec. 4.4, diamond mining.

Original: However, to the best of our knowledge,there is no published work that operates in the full, unmodiﬁed human action space, which includesdrag-and-drop inventory management and item crafting.
Revised: However, to the best of our knowledge,there is no published work that operates in the full, unmodiﬁed human action space, which includesdrag-and-drop inventory management and item crafting.

Original:  Methods
Revised:  Methods

Original: Inverse Dynamics Models (IDM) VPT, illustrated in Figure 2, requires we ﬁrst collect a smallamount of labeled contractor data with which to train an inverse dynamics model p IDM ( a t | o 1 ...T ) ,which seeks to minimize the negative log-likelihood of an action at timestep t given a trajectory of Tobservations o t : t 2 [1 ...T ] .
Revised: Inverse Dynamics Models (IDM) VPT, illustrated in Figure 2, requires we ﬁrst collect a smallamount of labeled contractor data with which to train an inverse dynamics model p IDM ( a t | o 1 ...T ) , which seeks to minimize the negative log-likelihood of an action at timestep t given a trajectory of Tobservations o t : t 2 [1 ...T ] .

Original: In contrast to an imitation learning policy, the IDM can be non-causal,meaning its prediction for a t can be a function of both past and future events , i.e. o t 0 >t .
Revised: In contrast to an imitation learning policy, the IDM can be non-causal,meaning its prediction for a t can be a function of both past and future events , i.e. o t 0 >t .

Original: Compared tothe behavioral cloning objective of modeling the distribution of human intent given past frames only,we hypothesize that inverting environment dynamics is easier and more data efﬁcient to learn.
Revised: Compared tothe behavioral cloning objective of modeling the distribution of human intent given past frames only,we hypothesize that inverting environment dynamics is easier and more data efﬁcient to learn.

Original: Indeed,
Revised: Indeed,

Original: Sec. 4.1 will show that the IDM objective is much easier to learn, and furthermore Sec.
Revised: Sec. 4.1 will show that the IDM objective is much easier to learn, and furthermore Sec.

Original: 4.6 will showthat with very little labeled data (as few as 100 hours) we can train a fairly accurate IDM.
Revised: 4.6 will showthat with very little labeled data (as few as 100 hours) we can train a fairly accurate IDM.

Original: This IDM can be used to label online videos, providing the large amount of data required for the harder task ofbehavioral cloning.
Revised: This IDMcan be used to label online videos, providing the large amount of data required for the harder task ofbehavioral cloning.

Original: See appendices D and B for IDM training and data collection details.
Revised: See appendices D and B for IDM training and data collection details.

Original: Data Filtering We gather a large dataset of Minecraft videos by searching the web for relatedkeywords (Appendix A).
Revised: Data Filtering We gather a large dataset of Minecraft videos by searching the web for relatedkeywords (Appendix A).

Original: Online videos often (1) include overlaid artifacts, such as a video feedof the player’s face, channel logos, watermarks, etc., (2) are collected from platforms other thana computer with different gameplay, or (3) are from different game modes, e.g. in Minecraft weonly want "survival mode" where players start from scratch and must gather or craft all their items.
Revised: Online videos often (1) include overlaid artifacts, such as a video feedof the player’s face, channel logos, watermarks, etc., (2) are collected from platforms other thana computer with different gameplay, or (3) are from different game modes, e.g. in Minecraft weonly want "survival mode" where players start from scratch and must gather or craft all their items.

Original: We call data “clean” if it does not contain visual artifacts and is from survival mode, and call allother data “unclean.”
Revised: We call data “clean” if it does not contain visual artifacts and is from survival mode, and call allother data “unclean.”

Original: With enough data, a large enough model, and enough training compute, a BCmodel trained on both unclean and clean videos would likely still perform well in a clean Minecraftenvironment.
Revised: With enough data, a large enough model, and enough training compute, a BCmodel trained on both unclean and clean videos would likely still perform well in a clean Minecraftenvironment.

Original: However, for simplicity and training compute efﬁciency, we choose to ﬁlter out uncleansegments of video (note that a video may contain both clean and unclean segments).
Revised: However, for simplicity and training compute efﬁciency, we choose to ﬁlter out uncleansegments of video (note that a video may contain both clean and unclean segments).

Original: We do this bytraining a model to ﬁlter out unclean segments using a small dataset (8800) of images sampled fromonline videos labeled by contractors as clean or unclean.
Revised: We do this bytraining a model to ﬁlter out unclean segments using a small dataset (8800) of images sampled fromonline videos labeled by contractors as clean or unclean (Appendix A.2).

Original: We did not tune this process as it is fairlystandard; see Appendix A.2 for more details and ablations showing data cleaning is beneﬁcial.

Original: VPT Foundation Model We train a foundation model with standard behavioral cloning, i.e.minimizing the negative log-likelihood of actions predicted by the IDM on clean data.
Revised: VPT Foundation Model We train a foundation model with standard behavioral cloning, i.e.minimizing the negative log-likelihood of actions predicted by the IDM on clean data.

Original: For a particulartrajectory of length T we minimize
Revised: For a particulartrajectory of length T we minimize

Original: [Equation]
Revised: [Equation]

Original: ) , where a

Original: [Equation]

Original: [Equation] [Equation] [Equation] 161 As we will see in the following sections, this model exhibits nontrivial zero-shot behavior and can beﬁne-tuned with both imitation learning and RL to perform even more complex skills.
Revised: As we will see in the following sections, this model exhibits nontrivial zero-shot behavior and can beﬁne-tuned with both imitation learning and RL to perform even more complex skills.

Original:  Results
Revised:  Results

Original: 4.1 Performance of the Inverse Dynamics Model
Revised: 4.1 Performance of the Inverse Dynamics Model

Original: The IDM architecture is comprised primarily of a temporal convolution layer, a ResNet 63 imageprocessing stack, and residual unmasked attention layers, from which the IDM simultaneouslypredicts keypresses and mouse movements (see Appendix D for IDM architecture and trainingdetails).
Revised: The IDM architecture is comprised primarily of a temporal convolution layer, a ResNet 62 imageprocessing stack, and residual unmasked attention layers, from which the IDM simultaneouslypredicts keypresses and mouse movements (see Appendix D for IDM architecture and trainingdetails).

Original: A key hypothesis behind our work is that IDMs can be trained with a relatively small amountof labeled data.
Revised: A key hypothesis behind our work is that IDMs can be trained with a relatively small amountof labeled data.

Original: While more data improves both mouse movement and keypress predictions, our best
Revised: While more data improves both mouse movement and keypress predictions, our best

Original: IDM trains on only 1962 hours of data (compared to the ⇠ 70k hours of clean data we collected fromthe internet) and achieves 90.6% keypress accuracy and a 0.97 R 2 for mouse movements evaluatedon a held-out validation set of contractor-labeled data (Figure 3 left).
Revised: IDM trains on only 1962 hours of data (compared to the ⇠ 70k hours of clean data we collected fromthe internet) and achieves 90.6% keypress accuracy and a 0.97 R 2 for mouse movements evaluatedon a held-out validation set of contractor-labeled data (Figure 3 left).

Original: Figure 3 (right) validates our hypothesis that IDMs are far more data efﬁcient than BC models, likelybecause inverting environment mechanics is far easier than modeling the entire distribution of humanbehavior.
Revised: Figure 3 (right) validates our hypothesis that IDMs are far more data efﬁcient than BC models, likelybecause inverting environment mechanics is far easier than modeling the entire distribution of humanbehavior.

Original: The IDM is two orders of magnitude more data efﬁcient than a BC model trained on thesame data and improves more quickly with more data.
Revised: The IDM is two orders of magnitude more data efﬁcient than a BC model trained on the same data and improves more quickly with more data.

Original: This evidence supports our hypothesis that itis more effective to use contractor data within the VPT pipeline by training an IDM than it is to traina foundation model from contractor data directly (Sections 4.5 and 4.6 provide additional evidence).
Revised: This evidence supports the hypothesis that it ismore effective to use contractor data within the VPT pipeline by training an IDM than it is to train a [Figure] foundation model from contractor data directly (Sections 4.5 and 4.6 provide additional evidence).

Original: Due to their data efﬁciency, training an IDM uses a negligible fraction of the overall compute neededto train a VPT model.
Revised: [Figure]

Original: 4.2 VPT Foundation Model Training and Zero-Shot Performance
Revised: 4.2 VPT Foundation Model Training and Zero-Shot Performance

Revised: [Figure]

Original: We now explore the emergent behavior learned by a behavioral cloning policy trained on an extremelylarge, but noisy, internet dataset labeled with our IDM.
Revised: We now explore the emergent behavior learned by a behavioral cloning policy trained on an extremelylarge, but noisy, internet dataset labeled with our IDM.

Original: To collect the unlabeled internet dataset, we searched for publicly available videos of Minecraft play with search terms such as “minecraftsurvival for beginners.”
Revised: To collect the unlabeled internet dataset,we searched for publicly available videos of Minecraft play with search terms such as “minecraftsurvival for beginners.”

Original: These searches resulted in ⇠ 270k hours of video, which we ﬁltered down to“clean” video segments yielding an unlabeled dataset of ⇠ 70k hours, which we refer to as web_clean(Appendix A has further details on data scraping and ﬁltering).
Revised: These searches resulted in ⇠ 270k hours of video, which we ﬁltered down to“clean” video segments yielding an unlabeled dataset of ⇠ 70k hours, which we refer to as web_clean(Appendix A has further details on data scraping and ﬁltering).

Original: We then generated pseudo-labelsfor web_clean with our best IDM (Section 3) and then trained the VPT foundation model withbehavioral cloning.
Revised: We then generated pseudo-labelsfor web_clean with our best IDM (Section 3) and then trained the VPT foundation model withbehavioral cloning.

Original: Preliminarymodel scaling experiments suggested that our model could beneﬁtfrom 30 epochs of training and that a 0.5 billion parameter model was required to stay in the efﬁcientlearning regime 64 for that training duration (Appendix H shows results comparing model size and thebeneﬁt of scaling to 0.5B parameters), which took ⇠ 9 days on 720 V100 GPUs.
Revised: Preliminary experiments suggested that our model could beneﬁt from 30 epochsof training and that a 0.5 billion parameter model was required to stay in the efﬁcient learningregime 63 for that training duration (Appendix H), which took ⇠ 9 days on 720 V100 GPUs.

Original: We evaluate our models by measuring validation loss (Fig. 4, left) and rolling them out in the
Revised: We evaluate our models by measuring validation loss (Fig. 4, left) and rolling them out in the

Original: Minecraft environment.
Revised: Minecraft environment.

Original: Unless otherwise noted, in all environment evaluations we spawn agents in astandard survival mode game where they play for 60 minutes, i.e. 72000 consecutive actions, and weplot the mean and shade the standard error of the mean for various game statistics such as craftingand collection rates (Fig. 4, right).
Revised: Unless otherwise noted, in all environment evaluations we spawn agents in astandard survival mode game where they play for 60 minutes, i.e. 72000 consecutive actions, and weplot the mean and shade the standard error of the mean for various game statistics such as craftingand collection rates (Fig. 4, right).

Original: The VPT foundation model quickly learns to chop down treesto collect logs, a task we found near impossible for an RL agent to achieve with the native humaninterface (Sec. 4.4).
Revised: The VPT foundation model quickly learns to chop down treesto collect logs, a task we found near impossible for an RL agent to achieve with the native humaninterface (Sec. 4.4).

Original: It also learns to craft those logs into wooden planks and then use those planksto craft a crafting table, which are required to unlock most other technology in the game and take ahuman proﬁcient in Minecraft approximately 50 seconds (970 consecutive actions) to collect.
Revised: It also learns to craft those logs into wooden planks and then use those planksto craft a crafting table, which are required to unlock most other technology in the game and take ahuman proﬁcient in Minecraft approximately 50 seconds (970 consecutive actions) to collect.

Original: Whilethese behaviors are fairly complex in the native human action space, the VPT foundation model craftsthese items at a rate far below that of our proﬁcient contractors, e.g. on average our contractors craft5.44 crafting tables in 60 minutes of play versus 0.19 for the foundation model.
Revised: Whilethese behaviors are fairly complex in the native human action space, the VPT foundation model craftsthese items at a rate far below that of our proﬁcient contractors, e.g. on average our contractors craft5.44 crafting tables in 60 minutes of play versus 0.19 for the foundation model.

Original: The model also craftsa non-negligible amount of wooden sticks, which are required to make wooden tools; collects variousﬂowers and crafts dyes from them; kills zombies that appear during the night; hunts wild animals;collects various berries and mushrooms and eats them; and ﬁnds game-generated villages from whichto collect various rare items from chests.
Revised: The model also craftsa non-negligible amount of wooden sticks, which are required to make wooden tools; collects variousﬂowers and crafts dyes from them; kills zombies that appear during the night; hunts wild animals;collects various berries and mushrooms and eats them; and ﬁnds game-generated villages from whichto collect various rare items from chests.

Original: The model also learned to navigate uneven terrain, swim,and pillar jump, which involves the agent repeatedly jumping and quickly placing a block below itselfsuch that it climbs upward by making a pillar.
Revised: The model also learned to navigate uneven terrain, swim,and pillar jump, which involves the agent repeatedly jumping and quickly placing a block below itselfsuch that it climbs upward by making a pillar.

Original: (i)
Revised: (i)

Original: While training and validation loss decrease healthily over training (Fig. 4, left), loss on our contractordataset (which the VPT model does not train on) begins increasing after 7 epochs.
Revised: While training and validation loss decrease healthily over training (Fig. 4, left), loss on our contractordataset (which the VPT model does not train on) begins increasing after 7 epochs.

Original: Contractor datacould be out-of-distribution because our contractors may have a different distribution of play orbecause there is some impactful visual domain shift compared to videos from the web, and we providesome evidence for this phenomenon in Appendix H. While one could have expected this would bepredictive of declining evaluation performance, we do not see notable game statistics from the VPTfoundation model rollouts (Figure 4, right) decrease over training, and in the next section we showthat BC ﬁne-tuning performance continually improves as the VPT foundation model trains. 
Revised: Contractor datacould be out-of-distribution because our contractors may have a different distribution of play orbecause there is some impactful visual domain shift compared to videos from the web. While one could have expected this would be predictive of declining evaluation performance, we do not seenotable game statistics from the VPT foundation model rollouts (Figure 4, right) decrease overtraining, and in the next section we show that BC ﬁne-tuning performance continually improves as the VPT foundation model trains. We provide more insight into this curious phenomenon in Appendix H.

Original: 4.3 Fine-Tuning with Behavioral Cloning
Revised: 4.3 Fine-Tuning with Behavioral Cloning

Original: Foundation models are designed to have a broad behavior proﬁle and be generally capable across awide variety of tasks.
Revised: Foundation models are designed to have a broad behavior proﬁle and be generally capable across awide variety of tasks.

Original: To incorporate new knowledge or allow them to specialize on a narrower task distribution, it is common practice to ﬁne-tune these models to smaller, more speciﬁc datasets.
Revised: To incorporate new knowledge or allow them to specialize on a narrower taskdistribution, it is common practice to ﬁne-tune these models to smaller, more speciﬁc datasets.

Original: The
Revised: The

Original: VPT foundation model trained on the broad web_clean dataset had nontrivial zero-shot performance;it was able to craft a crafting table yet unable to go past this in the technology tree.
Revised: VPT foundation model trained on the broad web_clean dataset had nontrivial zero-shot performance;it was able to craft a crafting table yet unable to go past this in the technology tree.

Original: As a casestudy into BC ﬁne-tuning, we attempt to improve the VPT foundation model’s ability to collectand craft these “early game” items by ﬁne-tuning to two narrower datasets targeted at Minecraftbehavior within the ﬁrst few minutes of players starting in a fresh world.
Revised: As a casestudy into BC ﬁne-tuning, we attempt to improve the VPT foundation model’s ability to collectand craft these “early game” items by ﬁne-tuning to two narrower datasets targeted at Minecraftbehavior within the ﬁrst few minutes of players starting in a fresh world.

Original: In the ﬁrst dataset,contractor_house , contractors have 10 minutes to build a basic house from scratch using primarilywood, sand, and dirt.
Revised: In the ﬁrst dataset,contractor_house , contractors have 10 minutes to build a basic house from scratch using primarilywood, sand, and dirt.

Original: Collecting contractor data can be difﬁcult and expensive, so we also construct adataset earlygame_keyword by searching for videos online with descriptions that match keywordssuch as “new world”, “let’s play episode 1”, etc.; this is a subset of web_clean and is labeled withthe IDM.
Revised: Collecting contractor data can be difﬁcult and expensive, so we also construct adataset earlygame_keyword by searching for videos online with descriptions that match keywordssuch as “new world”, “let’s play episode 1”, etc.; this is a subset of web_clean and is labeled withthe IDM.

Original: See Appendix B.4 and A.3 for full descriptions of both datasets.
Revised: See Appendix B.4 and A.3 for full descriptions of both datasets.

Original: [Figure]
Revised: [Figure]

Original: Fine-tuning to earlygame_keyword results in a large boost compared to the zero-shot foundationmodel: 2.5x more crafting tables, 6.1x more planks, 4.3x more logs, and 5.5x more crafting overall(Fig. 5).
Revised: Fine-tuning to earlygame_keyword results in a large boost compared to the zero-shot foundationmodel: 2.5x more crafting tables, 6.1x more planks, 4.3x more logs, and 5.5x more crafting overall(Fig. 5).

Original: However, when ﬁne-tuning to this dataset we did not see any new behaviors emerge,only a reﬁnement of existing skills.
Revised: However, when ﬁne-tuning to this dataset we did not see any new behaviors emerge,only a reﬁnement of existing skills.

Original: We saw an even bigger improvement when ﬁne-tuning to thecontractor_house dataset: 213x more crafting tables, 59x more wooden planks, 7x more logs,and 59x more crafting over all.
Revised: We saw an even bigger improvement when ﬁne-tuning to thecontractor_house dataset: 213x more crafting tables, 59x more wooden planks, 7x more logs,and 59x more crafting over all.

Original: In addition, we saw the emergence of crafting wooden tools, whichrequires placing a crafting table on the ground, opening it to reveal a new crafting interface, and thenusing it to craft wooden tools.
Revised: In addition, we saw the emergence of crafting wooden tools, whichrequires placing a crafting table on the ground, opening it to reveal a new crafting interface, and thenusing it to craft wooden tools.

Original: This entire sequence takes a proﬁcient human player a median of 1.2minutes (1390 consecutive actions) to accomplish.
Revised: This entire sequence takes a proﬁcient human player a median of 1.2minutes (1390 consecutive actions) to accomplish.

Original: The model goes further and collects cobblestone,which requires a wooden pickaxe to mine, and crafts stone tools, requiring it to again use a craftingtable; this takes a proﬁcient human player a median of 2.3 minutes (2790 consecutive actions).
Revised: The model goes further and collects cobblestone,which requires a wooden pickaxe to mine, and crafts stone tools, requiring it to again use a craftingtable; this takes a proﬁcient human player a median of 2.3 minutes (2790 consecutive actions).

Original: Wealso saw this model more frequently raiding villages that randomly spawn in the game, huntinganimals for food, in addition to many behaviors we saw performed by the foundation model.
Revised: Wealso saw this model more frequently raiding villages that randomly spawn in the game, huntinganimals for food, in addition to many behaviors we saw performed by the foundation model.

Original: (ii)
Revised: (ii)

Original: Despite the foundation model’s zero-shot rollout performance plateauing 1/3 into training (Fig.
Revised: Despite the foundation model’s zero-shot rollout performance plateauing 1/3 into training (Fig.

Original: 4,right), ﬁne-tuning performance does continue to increase throughout foundation model training(Fig.
Revised: 4,right), ﬁne-tuning performance does continue to increase throughout foundation model training(ii)

Revised: Sample Videos: https://www.youtube eWUiKm

Original:  5, right).
Revised: (Fig. 5, right).

Original: Additionally, there is a stark difference in performance when training from scratch vs.ﬁne-tuning from the VPT foundation model (Fig.
Revised: Additionally, there is a stark difference in performance when training from scratch vs.ﬁne-tuning from the VPT foundation model (Fig.

Original: 5 right, comparing the left and rightmost points).
Revised: 5 right, comparing the left and rightmost points).

Revised: [Figure]

Original: 4.4 Fine-Tuning with Reinforcement Learning
Revised: 4.4 Fine-Tuning with Reinforcement Learning

Original:  To demonstrate the efﬁcacy of RL ﬁne-tuning, we chose the challenging goal of obtaining a diamondpickaxe within 10 minutes starting from a fresh Minecraft survival world.
Revised: [Figure] To demonstrate the efﬁcacy of RL ﬁne-tuning, we chose the challenging goal of obtaining a diamondpickaxe within 10 minutes starting from a fresh Minecraft survival world.

Original: Doing so involves acquiringa sequence of difﬁcult-to-obtain items that require complex skills like mining, inventory management,crafting with and without a crafting table, tool use, operating a furnace, and mining at the lowestdepths, where many hazards like enemies and lava exist (Fig. 6).
Revised: Doing so involves acquiringa sequence of difﬁcult-to-obtain items that require complex skills like mining, inventory management,crafting with and without a crafting table, tool use, operating a furnace, and mining at the lowestdepths, where many hazards like enemies and lava exist (Fig. 6).

Original: Adding to the difﬁculty, progress(ii)
Revised: Adding to the difﬁculty, progresscan be easily lost by dropping items, destroying items, or dying.

Original: Videos: = _eWUiKm [Figure]
Revised: Obtaining a diamond pickaxe more often than not takes a proﬁcient human over 20 minutes (24,000 actions).

Original: Episodes last for 
Revised: Agents are rewarded for each item obtained in the sequence, with lower rewards for items that have tobe collected in bulk and higher rewards for items near the end of the sequence. Agents are optimized with the phasic policy gradient 64 RL algorithm for ⇠ 1.3 million episodes

Original: See G.1 for  function and RL  details.
Revised: See Appendix G for reward function and RL training details.

Original: Due tocomputational constraints, RL experiments use a ⇠ 248 million parameter VPT model (Appendix H).
Revised: Due to computational constraints, RLexperiments use a ⇠ 248 million parameter VPT model (Appendix H).

Original: A major problem when ﬁne-tuning with RL is catastrophic forgetting 66,67 because previously learnedskills can be lost before their value is realized.
Revised: A major problem when ﬁne-tuning with RL is catastrophic forgetting 65,66 because previously learnedskills can be lost before their value is realized.

Original: For instance, while our VPT foundation model neverexhibits the entire sequence of behaviors required to smelt iron zero-shot, it did train on examples ofplayers smelting with furnaces.
Revised: For instance, while our VPT foundation model neverexhibits the entire sequence of behaviors required to smelt iron zero-shot, it did train on examples ofplayers smelting with furnaces.

Original: It therefore may have some latent ability to smelt iron once the manyprerequisites to do so have been performed.
Revised: It therefore may have some latent ability to smelt iron once the manyprerequisites to do so have been performed.

Original: To combat the catastrophic forgetting of latent skillssuch that they can continually improve exploration throughout RL ﬁne-tuning, we add an auxiliary
Revised: To combat the catastrophic forgetting of latent skillssuch that they can continually improve exploration throughout RL ﬁne-tuning, we add an auxiliary

Original: Kullback-Leibler (KL) divergence loss between the RL model and the frozen pretrained policy.
Revised: Kullback-Leibler (KL) divergence loss between the RL model and the frozen pretrained policy.

Original: Training from a randomly initialized policy fails to achieve almost any reward, underscoring howhard an exploration challenge the diamond pickaxe task is for RL in the native human action space(Fig.
Revised: Training from a randomly initialized policy fails to achieve almost any reward, underscoring howhard an exploration challenge the diamond pickaxe task is for RL in the native human action space(Fig.

Original: 7a).
Revised: 7a).

Original: The model never learns toreliably collect logs, typically the ﬁrst of many steps to obtaininga diamond pickaxe (Fig. 7b).
Revised: Only at the very end of the training budget does this model learn to collect logs, typicallythe ﬁrst of many steps to obtaining a diamond pickaxe (Fig. 7b).

Original: RL ﬁne-tuning from the VPT foundation model does substantiallybetter (Fig. 7a),  learningeverything up to mining iron ore and crafting furnaces. (Fig. 7c).
Revised: RL ﬁne-tuning from the VPTfoundation model does substantially better (Fig. 7a), quickly learning to collect logs and craft planks,sticks, and crafting tables (Fig. 7c).

Original: However,this agent fails at smelting an iron ingot, the next item required to get further into the tech tree, likelybecause the zero-shot probability that the VPT foundation model smelts an iron ingot is too low, evenwhen given the prerequisite materials.
Revised: However, this agent fails at crafting a wooden pickaxe, the nextitem required to get further into the tech tree, likely because it gets stuck in a local reward optimum.

Original: Results further improve by ﬁrst BC ﬁne-tuning the VPT Foundation Model to theearlygame_keyword dataset (the early-game model , Sec. 4.3) and then ﬁne-tuning with RL (Fig. 7a), which in preliminary experiments we found to perform better than ﬁrst ﬁne-tuning tocontractor_house followed by ﬁne-tuning with RL (Appendix G.2).
Revised: Results dramatically improve by ﬁrst BC ﬁne-tuning the VPT Foundation Model to theearlygame_keyword dataset (the early-game model , Sec. 4.3) and then ﬁne-tuning with RL(Fig. 7a), which in preliminary experiments we found to perform better than ﬁrst ﬁne-tuning tocontractor_house followed by ﬁne-tuning with RL (Appendix G.1).

Original: The three-phase training(pretraining, BC ﬁne-tuning, and then RL ﬁne-tuning) succeeds in learning extremely difﬁcult tasks:it achieves over 80% reliability on iron pickaxes, almost 20% reliability on collecting diamonds, and2 .
Revised: The 3-phase training (pretraining, BC ﬁne-tuning, and then RL ﬁne-tuning) succeeds in learning extremely difﬁcult tasks: itachieves over 80% reliability on iron pickaxes, over 15% reliability on collecting diamonds, and over1 .

Original: 5% reliability on obtaining a diamond pickaxe (Fig. 7d).
Revised: 5% reliability on obtaining a diamond pickaxe (Fig. 7d).

Original: For comparison, human players giventhe objective of obtaining a diamond pickaxe collect these items in 57% , 15% , and 12% of episodes,respectively, meaning our model is human-level for crafting iron pickaxes and mining diamonds.
Revised: For comparison, human players giventhe objective of obtaining a diamond pickaxe collect these items in 57% , 15% , and 12% of episodes,respectively, meaning our model is human-level for crafting iron pickaxes and mining diamonds.

Original: Others have managed to obtain diamonds with ⇠ 0 .
Revised: Others have managed to obtain diamonds with ⇠ 0 .

Original: 1% reliability in 15 minutes 33,34 but always with asimpliﬁed action space designed to ease exploration.
Revised: 1% reliability in 15 minutes 33,34 but always with asimpliﬁed action space designed to ease exploration.

Original: To the best of our knowledge, we are the ﬁrst toreport non-zero success rates on crafting a diamond pickaxe .
Revised: To the best of our knowledge, we are the ﬁrst toreport non-zero success rates on crafting a diamond pickaxe .

Original: Qualitatively, the model developeduseful skills for diamond mining, such as efﬁcient mining patterns, cave exploration, returning topreviously placed objects like crafting tables, and advanced techniques like using wooden pickaxesas fuel when moving on to iron tools.
Revised: Qualitatively, the model developeduseful skills for diamond mining, such as efﬁcient mining patterns, cave exploration, returning topreviously placed objects like crafting tables, and advanced techniques like using wooden pickaxesas fuel when moving on to iron tools.

Original: (iii)
Revised: (iii)(iii) Videos found at https [Figure] When training the early-game model without a KL loss to the original policy ( No KL loss ) progress stalls episodes, suggesting that the skills necessary to make further progress have been catastrophically forgotten a randomly initialized

Original: Finally, we validated the importance of the KL loss to the pretrained model during RL ﬁne-tuning.
Revised: Finally, we validated the importance of the KL loss to the pretrained model during RL ﬁne-tuning.

Original: The treatment without a KL loss obtains only items early in the sequence (logs, planks, sticks, andcrafting tables) limiting its reward (Fig. 7a).
Revised: The treatment without a KL loss obtains only items early in the sequence (logs, planks, sticks, andcrafting tables) limiting its reward (Fig. 7a).

Original: This failure to progress further into the sequence islikely because, while the initial skills of chopping logs and crafting planks are being learned with RL,subsequent skills like crafting a wooden pickaxe are lost due to catastrophic forgetting.
Revised: This failure to progress further into the sequence islikely because, while the initial skills of chopping logs and crafting planks are being learned with RL,subsequent skills like crafting a wooden pickaxe are lost due to catastrophic forgetting.

Original: 4.5 Data Scaling Properties of the Foundation Model
Revised: 4.5 Data Scaling Properties of the Foundation Model

Original: [Figure]
Revised: [Figure]

Revised: compute limitations this was performed

Original: In this section we validate a core hypothesis behind this work: that it is far more effective to uselabeled contractor data to train an IDM within the VPT method than it is to directly train a BCfoundation model from that same small contractor dataset.
Revised: In this section we validate a core hypothesis behind this work: that it is far more effective to uselabeled contractor data to train an IDM within the VPT method than it is to directly train a BCfoundation model from that same small contractor dataset.

Original: If we could cheaply collect a labeledcontractor dataset of a similar order of magnitude as web_clean , then this would not be important;however, collecting that scale of data would have cost millions of dollars.
Revised: If we could cheaply collect a labeledcontractor dataset of a similar order of magnitude as web_clean , then this would not be important;however, collecting that scale of data would have cost millions of dollars.

Original: Figure 8 comparesfoundation models trained on increasing orders of magnitude of data from 1 hour up to the full ⇠ 70kweb_clean dataset.
Revised: Figure 8 comparesfoundation models trained on increasing orders of magnitude of data from 1 hour up to the full ⇠ 70kweb_clean dataset.

Original: Foundation models trained up to and including 1k hours are trained on the IDMcontractor data, and those trained on 5k hours and above are trained on subsets of web_clean , whichdoes not contain any IDM contractor data.
Revised: Foundation models trained up to and including 1k hours are trained on the IDMcontractor data, and those trained on 5k hours and above are trained on subsets of web_clean , whichdoes not contain any IDM contractor data.

Original: Scaling training data increases log collection, mining, andcrafting capabilities.
Revised: Scaling training data increases log collection, mining, andcrafting capabilities.

Original: The zero-shot model only begins to start crafting crafting tables at over 5000hours of training data.
Revised: The zero-shot model only begins to start crafting crafting tables at over 5000hours of training data.

Original: When ﬁne-tuning each foundation model to contractor_house , we see thatcrafting rates for crafting tables and wooden tools increase by orders of magnitude when using theentire ⇠ 70k hour web_clean dataset.
Revised: When ﬁne-tuning each foundation model to contractor_house , we see thatcrafting rates for crafting tables and wooden tools increase by orders of magnitude when using the entire ⇠ 70k hour web_clean dataset.

Original: We furthermore only see the emergence of crafting stone toolsat the largest data scale.
Revised: We furthermore only see the emergence of crafting stone toolsat the largest data scale.

Original: 4.6 Effect of Inverse Dynamics Model Quality on Behavioral Cloning
Revised: 4.6 Effect of Inverse Dynamics Model Quality on Behavioral Cloning

Revised: [Figure]

Original: This section investigates how downstream BC performance is affected by IDM quality.
Revised: This section investigates how downstream BC performance is affected by IDM quality.

Original: We train IDMs on increasingly larger datasets and use each to independently label the earlygame_keyworddataset (this smaller dataset  was chosen due to a limited compute budget).
Revised: We train IDMs on increasingly largerdatasets and use each to independently labelthe earlygame_keyword dataset (this smallerdataset was chosen due to a limited compute budget).

Original: We then train a BC modelfrom scratch on each dataset and report game statistics for each model as a function of IDM contractordataset size (Fig. 9).[Figure]
Revised: We then train a BC model from scratch oneach dataset and report game statistics for eachmodel as a function of IDM contractor datasetsize (Fig. 9).

Original: IDMs trained on at least 10 hours of data are required for any crafting, and the crafting rate increasesquickly up until 100 hours of data, after which there are few to no gains and differences are likelydue to noise.
Revised: IDMs trained on at least 10 hours of data arerequired for any crafting, and the crafting rateincreases quickly up until 100 hours of data, after which there are few to no gains and differences arelikely due to noise.

Original: Similarly, crafting tables are only crafted after 50 or more hours of IDM data, and againgains plateau after 100 hours.
Revised: Similarly, crafting tables are only crafted after 50 or more hours of IDM data, andagain gains plateau after 100 hours.

Original: While in all previous experiments we use our best IDM trained on1962 hours of data, these results suggest we could reduce that number to as low as 100 hours.
Revised: While in all previous experiments we use our best IDM trainedon 1962 hours of data, these results suggest we could reduce that number to as low as 100 hours.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: [Figure]

Original: trained from scratch on the earlygame_ dataset labeled with IDMs that were trained on increasing amounts of contractor data.

Original: The results presented in this paper help pavethe path to utilizing the wealth of unlabeleddata on the web formany sequential decisiondomains.
Revised: The results presented in this paper help pave the path to utilizing the wealth of unlabeled data on theweb for sequential decision domains.

Original: Compared torepresentation learningmethods, e.g. generative video modeling, VPToffers the exciting possibility of directly learning to act during pretraining and using theselearned behavioral priors as extremely effectiveexploration priors for RL.
Revised: Compared to generative video modeling or contrastive methodsthat would only yield representational priors, VPT offers the exciting possibility of directly learningto act during pretraining and using these learned behavioral priors as extremely effective explorationpriors for RL.

Original: VPT could even bean effective representation learning method fordownstream tasks that do not require acting, e.g.video captioning, because arguably the most important information in any given scene would be present in features trained to correctly predict thedistribution over future human actions.
Revised: VPT could even be a better general representation learning method even when thedownstream task is not learning to act in that domain—for example, ﬁne-tuning to explain what ishappening in a video—because arguably the most important information in any given scene would bepresent in features trained to correctly predict the distribution over future human actions.

Original: We leave this intriguing direction to future work.
Revised: We leavethis intriguing direction to future work.

Original: Future work could improve results with more data (we estimate we could collect >1M hours) andlarger, better-tuned models.
Revised: Future work could improve results with more data (we estimate we could collect >1M hours, i.e.> $20M worth, of clean data) and larger, better-tuned models.

Original: Our internet data was fairly noisy and varied (players choose their owngraphics settings); we hope future work will investigate even noisier sources of data, as well as howto use both ﬁrst and third person demonstrations.

Original: Furthermore, all  models in this work conditionon past observations only; we cannot ask the modelto perform speciﬁc tasks.
Revised: Furthermore, all the models in thiswork condition on past observations only; we cannot ask the model perform speciﬁc tasks.

Original: Appendix I presentspreliminary experiments on conditioning our models on closed captions (text transcripts of speech invideos), showing they become weakly steerable; we believe this a rich direction for future research.
Revised: Appendix Ipresents preliminary experiments on conditioning our models on closed captions (text transcriptsof speech in videos), showing they become weakly steerable; we believe this a rich direction forfuture research.

Original: By deﬁnition behavioral priors must predict actions, and in this work we found this objective sufﬁcientto train capable agents; however, a fruitful direction could be incorporating auxiliary representationlearning objectives (e.g. contrastive losses, environment dynamics modeling, etc.) to reduce thesample complexity of both the IDM and foundation models.

Original: Similarly, it would be interesting to see if

Original: VPT could beneﬁt from pretraining its attention layers with a language modeling task as in Li et al.

Original: 68and Reid et al. 69 Loss was not consistently correlated with downstream evaluation metrics (Sec. 4.2),which often made progress slow.
Revised: Also, loss was not consistently correlated with downstream evaluation metrics(Sec. 4.2), which often made progress slow and hard-won.

Original: Another worthwhile future direction would be to investigate thecorrelation between various training metrics and downstream evaluations.
Revised: Another fruitful future direction would beto investigate the correlation between various training metrics and downstream evaluations.

Original: For RL ﬁne-tuning we only experimented with a standard policy gradient based RL algorithm

Original: (PPG);an interesting future direction would be to investigate how well VPT can be combined with other

Original: RL algorithms, e.g. off-policy or model based.

Original: Furthermore, we showed the efﬁcacy of ﬁne-tuning VPT with RL using a very difﬁcult, albeit handcrafted, reward function aimed at crafting diamondtools. We hope future work will combine VPT with methods that can generate more generic rewardfunctions, e.g. natural language based reward functions as proposed in MineDojo 70 (released afterthis paper). Finally, while we do not anticipate any direct negative societal impacts from the modelstrained in this work, as VPT improves and expands to other domains it will be important to assess andmitigate harms that emerge with other forms of pretraining on internet datasets, such as emulatinginappropriate behavior.
Revised: Finally,while we do not anticipate any direct negative societal impacts from the models trained in this work,as VPT improves and expands to other domains it will be important to assess and mitigate harmsthat emerge with other forms of pretraining on internet datasets, such as emulating inappropriatebehavior.

Original: In conclusion, VPT extends the paradigm of training large and general purpose behavioral priorsto sequential decision domains that have commonly available unlabeled internet data.
Revised: In conclusion, VPT extends the paradigm of training large and general purpose behavioral priors fromfreely available internet-scale data to sequential decision domains.

Original: Our modelsexhibited impressive zero-shot behavior and, when ﬁne-tuned with RL, achieved an unprecedentedresult of crafting a diamond pickaxe in Minecraft (all the more difﬁcult given the human interface).
Revised: Our models exhibited impressivezero-shot behavior and, when ﬁne-tuned with RL, achieved an unprecedented result of crafting adiamond pickaxe in Minecraft (all the more difﬁcult given the human interface).

Original: We further showed that contractor data is far better used within the VPT pipeline than to train afoundation model directly and that only a small amount of contractor data (about $2000 USD) wasrequired to unlock massive amounts of unlabeled online data for use in BC.
Revised: We further showedthat contractor data is far better used within the VPT pipeline than to train a foundation model directlyand that only a small amount of contractor data (about $2000 USD) was required to unlock millionsof dollars worth of unlabeled online data for use in BC.

Original: Finally, learning withthe human keyboard and mouse interface is highly general and allows losslessly modeling the entiredistribution of human behavior.
Revised: Finally, learning with the human keyboardand mouse interface is highly general and allows losslessly modeling the entire distribution of humanbehavior.

Original: While we only experiment in Minecraft, we believe that VPT providesa general recipe for training behavioral priors in hard, yet generic, action spaces in any domain thathas a large amount of freely available unlabeled data, such as computer usage.
Revised: While we only experiment in Minecraft, we believe that VPT provides a general recipe fortraining behavioral priors in hard, yet generic, action spaces in any domain that has a large amount offreely available unlabeled data, such as computer usage.