Original: R3M: A Universal Visual Representation for Robot Manipulation
Revised: R3M: A Universal Visual Representation for Robot Manipulation

Original: Abstract: We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks.
Revised: Abstract: We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks.

Original: Concretely, we pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations.
Revised: Concretely, we pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations.

Original: The resulting representation, R3M, can be used as a frozen perception module for downstream policy learning.
Revised: The resulting representation, R3M, can be used as a frozen perception module for downstream policy learning.

Original: Across a suite of 12 simulated robot manipulation tasks, we find that R3M improves task success by over 20% compared to training from scratch and by over 10% compared to state-of-the-art visual representations like CLIP and
Revised: Across a suite of 12 simulated robot manipulation tasks, we find that R3M improves task success by over 20% compared to training from scratch and by over 10% compared to state-of-the-art visual representations like CLIP and

Original: MoCo.
Revised: MoCo.

Original: Furthermore, R3M enables a Franka Emika Panda arm to learn a range of manipulation tasks in a real, cluttered apartment given just 20 demonstrations.
Revised: Furthermore, R3M enables a Franka Emika Panda arm to learn a range of manipulation tasks in a real, cluttered apartment given just 20 demonstrations.

Original:  Introduction
Revised:  Introduction

Original: How do we train a robot to complete a manipulation task from images?
Revised: How do we train a robot to complete a manipulation task from images?

Original: A standard and widely used approach is to train an end-to-end model from scratch using data from the same domain [1].
Revised: A standard and widely used approach is to train an end-to-end model from scratch using data from the same domain [1].

Original: However, this can be prohibitively data intensive and severely limits generalization.
Revised: However, this can be prohibitively data intensive and severely limits generalization.

Original: In contrast, computer vision and natural language processing (NLP) have recently taken a major departure from this “tabula rasa” paradigm.
Revised: In contrast, computer vision and natural language processing (NLP) have recently taken a major departure from this “tabula rasa” paradigm.

Original: These fields have focused on using diverse, large-scale datasets to build reusable, pre-trained representations .
Revised: These fields have focused on using diverse, large-scale datasets to build reusable, pre-trained representations .

Original: Such models have become ubiquitous; for example, visual representations from ImageNet [2] can be reused for tasks like cancer detection [3], and pre-trained language embeddings like BERT [4] have been used for everything from medical coding [5] to visual question answering [6].
Revised: Such models have become ubiquitous; for example, visual representations from ImageNet [2] can be reused for tasks like cancer detection [3], and pre-trained language embeddings like BERT [4] have been used for everything from medical coding [5] to visual question answering [6].

Original: Such an equivalent of an ImageNet [2] or BERT [4] model for robotics, that can be readily downloaded and used for any downstream simulation or real-world manipulation task, has remained elusive.
Revised: Such an equivalent of an ImageNet [2] or BERT [4] model for robotics, that can be readily downloaded and used for any downstream simulation or real-world manipulation task, has remained elusive.

Original: Why have we struggled in building this universal representation for robotics?
Revised: Why have we struggled in building this universal representation for robotics?

Original: Our conjecture is that we haven’t converged on using the appropriate datasets for robotics.
Revised: Our conjecture is that we haven’t converged on using the appropriate datasets for robotics.

Original: Collecting large and diverse datasets of robots interacting with the physical world can be costly, even without human annotation.
Revised: Collecting large and diverse datasets of robots interacting with the physical world can be costly, even without human annotation.

Original: Recent attempts at creating such datasets [7, 8, 9, 10], consist of a limited number of tasks in at most a handful of different environments.
Revised: Recent attempts at creating such datasets [7, 8, 9, 10], consist of a limited number of tasks in at most a handful of different environments.

Original: This lack of diversity and scale makes it difficult to learn representations that are broadly applicable.
Revised: This lack of diversity and scale makes it difficult to learn representations that are broadly applicable.

Original: At the same time, the recent history of computer vision and NLP suggests an alternate route for robotics.
Revised: At the same time, the recent history of computer vision and NLP suggests an alternate route for robotics.

Original: The best representations in these fields did not arise out of task-specific and carefully curated datasets, but rather the use of abundant in-the-wild data [4, 11, 12, 13].
Revised: The best representations in these fields did not arise out of task-specific and carefully curated datasets, but rather the use of abundant in-the-wild data [4, 11, 12, 13].

Original: Analogously, for robotics and motor control, we have access to videos of humans interacting in semantically interesting ways with their environments [14, 15, 16].
Revised: Analogously, for robotics and motor control, we have access to videos of humans interacting in semantically interesting ways with their environments [14, 15, 16].

Original: This data is large and diverse, spanning scenes across the globe, and tasks ranging from folding clothes to cooking a meal.
Revised: This data is large and diverse, spanning scenes across the globe, and tasks ranging from folding clothes to cooking a meal.

Original: While the embodiment present in this data differs from most robots, prior work [17, 18] has found that such human video data can still be useful for learning reward functions.
Revised: While the embodiment present in this data differs from most robots, prior work [17, 18] has found that such human video data can still be useful for learning reward functions.

Original: Furthermore, domain gap has not been a major barrier for using pre-trained representations in traditional vision and
Revised: Furthermore, domain gap has not been a major barrier for using pre-trained representations in traditional vision and

Original: NLP tasks.
Revised: NLP tasks.

Original: In this backdrop, we ask the pertinent question: can visual representations pre-trained on diverse human videos enable efficient downstream learning of robotic manipulation skills?
Revised: In this backdrop, we ask the pertinent question: can visual representations pre-trained on diverse human videos enable efficient downstream learning of robotic manipulation skills?

Original: Ego4D Video + Language
Revised: Ego4D Video + Language

Original: [Figure]
Revised: [Figure]

Original: Robot
Revised: Robot

Original: [Figure]
Revised: [Figure]

Original: We hypothesize that a good representation for vision-based robotic manipulation consists of three components.
Revised: We hypothesize that a good representation for vision-based robotic manipulation consists of three components.

Original: First, it should contain information necessary for physical interaction, and thus should capture the temporal dynamics of the scene (i.e. how states might transition to other states).
Revised: First, it should contain information necessary for physical interaction, and thus should capture the temporal dynamics of the scene (i.e. how states might transition to other states).

Original: Second, it should have a prior over semantic relevance, and should focus on task relevant features like objects and their relationships.
Revised: Second, it should have a prior over semantic relevance, and should focus on task relevant features like objects and their relationships.

Original: Finally, it should be compact, and not include features irrelevant to the above criteria (e.g. backgrounds).
Revised: Finally, it should be compact, and not include features irrelevant to the above criteria (e.g. backgrounds).

Original: Towards satisfying these three criteria, we study a representation learning approach that combines (1) time contrastive learning [19] to learn a representation that captures temporal dynamics, (2) video-language alignment to capture semantically relevant features of the scene, and (3) L1 and L2 penalties to encourage sparsity.
Revised: Towards satisfying these three criteria, we study a representation learning approach that combines (1) time contrastive learning [19] to learn a representation that captures temporal dynamics, (2) video-language alignment to capture semantically relevant features of the scene, and (3) L1 and L2 penalties to encourage sparsity.

Original: Our experimental evaluation in Section 4. finds that all three components are important for training highly performant representations.
Revised: Our experimental evaluation in Section 4. finds that all three components are important for training highly performant representations.

Original: In this work we empirically demonstrate that representations pre-trained on diverse human video datasets like Ego4D [16] can enable efficient downstream policy learning for robotic manipulation.
Revised: In this work we empirically demonstrate that representations pre-trained on diverse human video datasets like Ego4D [16] can enable efficient downstream policy learning for robotic manipulation.

Original: Our core contribution is an artifact – the pre-trained vision model – that can be used readily in other work.
Revised: Our core contribution is an artifact – the pre-trained vision model – that can be used readily in other work.

Original: Concretely, we pre-train a r eusable r epresentation for r obotic m anipulation (R3M), which can be used as a frozen perception module for downstream policy learning in simulated and real robot manipulation tasks.
Revised: Concretely, we pre-train a r eusable r epresentation for r obotic m anipulation (R3M), which can be used as a frozen perception module for downstream policy learning in simulated and real robot manipulation tasks.

Original: We demonstrate this via extensive experimental results across three existing benchmark simulation environments (Adroit [20], Franka-Kitchen [21], and MetaWorld [22]) as well as real robot experiments in a cluttered apartment setting.
Revised: We demonstrate this via extensive experimental results across three existing benchmark simulation environments (Adroit [20], Franka-Kitchen [21], and MetaWorld [22]) as well as real robot experiments in a cluttered apartment setting.

Original: R3M features outperform a wide range of visual representations like CLIP [12], (supervised)
Revised: R3M features outperform a wide range of visual representations like CLIP [12], (supervised)

Original: ImageNet [2], MoCo [23, 24], and learning from scratch by over 10% when evaluated across 12 tasks, 9 viewpoints, and 3 different simulation environments.
Revised: ImageNet [2], MoCo [23, 24], and learning from scratch by over 10% when evaluated across 12 tasks, 9 viewpoints, and 3 different simulation environments.

Original: On a Franka Emika Panda robot, R3M enables learning challenging tasks like putting lettuce in a pan and folding a towel with a 50+% average success rate, given less than 10 minutes of human demonstrations (see Figure 1), which is nearly double the success rate compared to CLIP features.
Revised: On a Franka Emika Panda robot, R3M enables learning challenging tasks like putting lettuce in a pan and folding a towel with a 50+% average success rate, given less than 10 minutes of human demonstrations (see Figure 1), which is nearly double the success rate compared to CLIP features.

Original: Overall, on the basis of these results, we believe that R3M has the potential to become a standard vision model for robot manipulation, which can be simply downloaded and used off-the-shelf for any robot manipulation task or environment.
Revised: Overall, on the basis of these results, we believe that R3M has the potential to become a standard vision model for robot manipulation, which can be simply downloaded and used off-the-shelf for any robot manipulation task or environment.

Original: See https: //sites.google.com/view/robot-r3m for pre-trained models and code.
Revised: See https: //sites.google.com/view/robot-r3m for pre-trained models and code.

Original:  Related Work
Revised:  Related Work

Original: Representation Learning for Robotics.
Revised: Representation Learning for Robotics.

Original: Our work is certainly not the first to study the problem of learning general representations for robotics.
Revised: Our work is certainly not the first to study the problem of learning general representations for robotics.

Original: One line of work focuses on learning representations from in-domain data, that is, using data from the target environment and task for training the representation.
Revised: One line of work focuses on learning representations from in-domain data, that is, using data from the target environment and task for training the representation.

Original: Such methods include contrastive learning with data augmentation [25, 26, 27, 28], dynamics prediction [29, 30], bi-simulation [31], temporal or goal distance [32, 33], or domain specific information [34].
Revised: Such methods include contrastive learning with data augmentation [25, 26, 27, 28], dynamics prediction [29, 30], bi-simulation [31], temporal or goal distance [32, 33], or domain specific information [34].

Original: However, because they are trained on data exclusively from the target domain and task, the learned representations fail to generalize and cannot be re-used to enable faster learning in unseen tasks and environments.
Revised: However, because they are trained on data exclusively from the target domain and task, the learned representations fail to generalize and cannot be re-used to enable faster learning in unseen tasks and environments.

Original: Recently, there has been growing interest in learning more general representations for motor control from large-scale out-of-domain data like images from the web.
Revised: Recently, there has been growing interest in learning more general representations for motor control from large-scale out-of-domain data like images from the web.

Original: This includes the use of CLIP, supervised MS-COCO, supervised ImageNet, MoCo ImageNet features, or data from different robots [35, 36, 37, 38, 23, 39].
Revised: This includes the use of CLIP, supervised MS-COCO, supervised ImageNet, MoCo ImageNet features, or data from different robots [35, 36, 37, 38, 23, 39].

Original: In contrast to prior work, we pre-train the representation using diverse human video and language data, as opposed to static frames and/or class labels.
Revised: In contrast to prior work, we pre-train the representation using diverse human video and language data, as opposed to static frames and/or class labels.

Original: Further, in our experimental evaluation, we observe that our pre-trained representation outperforms prior work significantly on a comprehensive evaluation suite.
Revised: Further, in our experimental evaluation, we observe that our pre-trained representation outperforms prior work significantly on a comprehensive evaluation suite.

Original: Concurrently, Xiao et al.
Revised: Concurrently, Xiao et al.

Original: [40] also explore the use of human interaction data to pre-train visual representations for motor control.
Revised: [40] also explore the use of human interaction data to pre-train visual representations for motor control.

Original: However their learned representation only uses static frames from these videos and does not utilize temporal or semantic information like R3M. Furthermore, our evaluation focuses on data efficient imitation learning, and enables real-world learning in cluttered environments with just ∼ 10 minutes of demonstration data.
Revised: However their learned representation only uses static frames from these videos and does not utilize temporal or semantic information like R3M. Furthermore, our evaluation focuses on data efficient imitation learning, and enables real-world learning in cluttered environments with just ∼ 10 minutes of demonstration data.

Original: Leveraging Human Videos for Robot Learning .
Revised: Leveraging Human Videos for Robot Learning .

Original: Several prior works have explored using human video data in robot learning, for example to acquire goals [41, 42, 43], to learn visual dynamics models [44, 45, 46, 47], or to learn representations and rewards [19, 48, 49, 50, 51, 52].
Revised: Several prior works have explored using human video data in robot learning, for example to acquire goals [41, 42, 43], to learn visual dynamics models [44, 45, 46, 47], or to learn representations and rewards [19, 48, 49, 50, 51, 52].

Original: However, these prior works typically focus on a small dataset of human videos closely resembling the robot environment.
Revised: However, these prior works typically focus on a small dataset of human videos closely resembling the robot environment.

Original: In contrast, our work leverages diverse human video data like Ego4D [16] to learn visual reusable visual representations that generalize broadly.
Revised: In contrast, our work leverages diverse human video data like Ego4D [16] to learn visual reusable visual representations that generalize broadly.

Original: Natural Language and Robotic Manipulation .
Revised: Natural Language and Robotic Manipulation .

Original: Prior works have explored the use of natural language in robot manipulation, primarily as a means of task specification [53, 54, 36, 55] or reward learning [56].
Revised: Prior works have explored the use of natural language in robot manipulation, primarily as a means of task specification [53, 54, 36, 55] or reward learning [56].

Original: In contrast, we use diverse human video data and language annotations to learn reusable visual representations for control.
Revised: In contrast, we use diverse human video data and language annotations to learn reusable visual representations for control.

Original: Prior work has also found visual representations informed by language, like CLIP [12], to be effective for control [36, 37].
Revised: Prior work has also found visual representations informed by language, like CLIP [12], to be effective for control [36, 37].

Original: Through empirical evaluations, we find that our R3M representation substantially outperforms CLIP for robot manipulation.
Revised: Through empirical evaluations, we find that our R3M representation substantially outperforms CLIP for robot manipulation.

Original: Learning from Diverse Robot Data.
Revised: Learning from Diverse Robot Data.

Original: Towards robots that generalize more broadly, there are a number of works that study how to scale up the size and diversity of data robots learn from.
Revised: Towards robots that generalize more broadly, there are a number of works that study how to scale up the size and diversity of data robots learn from.

Original: Many of these works focus on collecting and learning from robot data itself [57, 58, 7, 8, 9, 10, 59].
Revised: Many of these works focus on collecting and learning from robot data itself [57, 58, 7, 8, 9, 10, 59].

Original: However, these works often contain at most a handful of different environments, making generalization across a range of unseen scenes difficult.
Revised: However, these works often contain at most a handful of different environments, making generalization across a range of unseen scenes difficult.

Original: While we also aim to enable generalization by learning from diverse data, our focus is instead on (1) learning from human video data and hence a larger distribution of environments and tasks, and (2) pre-training a visual representation, as opposed to policies or models.
Revised: While we also aim to enable generalization by learning from diverse data, our focus is instead on (1) learning from human video data and hence a larger distribution of environments and tasks, and (2) pre-training a visual representation, as opposed to policies or models.

Original: Representation Learning from Videos.
Revised: Representation Learning from Videos.

Original: Finally, there is a rich literature of works that study learning image representations from videos [60, 61, 19, 62, 63, 64] outside of the context of robotics.
Revised: Finally, there is a rich literature of works that study learning image representations from videos [60, 61, 19, 62, 63, 64] outside of the context of robotics.

Original: Additionally, there are a number of works that use language to learn representations from videos [65, 66].
Revised: Additionally, there are a number of works that use language to learn representations from videos [65, 66].

Original: Critically, unlike all of these works, the main contribution of this work is not to propose a novel representation learning approach, but rather in studying if representations trained on diverse video and language of human interaction can enable more efficient learning of robotic manipulation.
Revised: Critically, unlike all of these works, the main contribution of this work is not to propose a novel representation learning approach, but rather in studying if representations trained on diverse video and language of human interaction can enable more efficient learning of robotic manipulation.

Original:  R3M: Reusable Representations for Robotic Manipulation
Revised:  R3M: Reusable Representations for Robotic Manipulation

Original: Our goal is to use diverse human video data to pre-train a single reusable visual representation for motor control, particularly robotic manipulation, that can enable efficient downstream learning in previously unseen environments and tasks.
Revised: Our goal is to use diverse human video data to pre-train a single reusable visual representation for motor control, particularly robotic manipulation, that can enable efficient downstream learning in previously unseen environments and tasks.

Original: In this section, we cover the different components of our approach, beginning by describing our problem formulation in Section 3.1, the data sources we use in Section 3.2, and our training objective in Section 3.3.
Revised: In this section, we cover the different components of our approach, beginning by describing our problem formulation in Section 3.1, the data sources we use in Section 3.2, and our training objective in Section 3.3.

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: Formally, we assume that we have access to a dataset D of N videos, where each video consists of a sequence of RGB frames [ I 0 , I 1 , ..., I T ] .
Revised: Formally, we assume that we have access to a dataset D of N videos, where each video consists of a sequence of RGB frames [ I 0 , I 1 , ..., I T ] .

Original: Additionally, we assume that each video is paired with a natural language description l , that describes what task is being completed in the video.
Revised: Additionally, we assume that each video is paired with a natural language description l , that describes what task is being completed in the video.

Original: From this data, our goal is to learn a single image encoder F ϕ , that maps images to a deterministic, continuous embedding, that is z
Revised: From this data, our goal is to learn a single image encoder F ϕ , that maps images to a deterministic, continuous embedding, that is z

Original: = F ϕ ( I ) .
Revised: = F ϕ ( I ) .

Original: Once trained, we want to be able to repeatedly reuse F for downstream policy learning.
Revised: Once trained, we want to be able to repeatedly reuse F for downstream policy learning.

Original: Specifically, the downstream problem will involve an agent sequentially
Revised: Specifically, the downstream problem will involve an agent sequentially

Original: [Figure]
Revised: [Figure]

Original: choosing actions given image observations I , and instead of using raw images as input, the agent will use the pre-trained F ϕ ( I ) as a state representation.
Revised: choosing actions given image observations I , and instead of using raw images as input, the agent will use the pre-trained F ϕ ( I ) as a state representation.

Original: 3.2 Data Sources
Revised: 3.2 Data Sources

Original: For our learned representation F ϕ to be useful in a wide range of downstream tasks and environments, it should (1) be trained on data that is diverse enough to facilitate generalization, and (2) provide a useful signal for features relevant to robotic manipulation.
Revised: For our learned representation F ϕ to be useful in a wide range of downstream tasks and environments, it should (1) be trained on data that is diverse enough to facilitate generalization, and (2) provide a useful signal for features relevant to robotic manipulation.

Original: One approach would be to be use natural images off the web (e.g. ImageNet [2]).
Revised: One approach would be to be use natural images off the web (e.g. ImageNet [2]).

Original: While diverse, these images tend to focus on one particular object, and do not capture an agent interacting with multiple objects in a scene.
Revised: While diverse, these images tend to focus on one particular object, and do not capture an agent interacting with multiple objects in a scene.

Original: Alternatively, data of humans interacting in the world [14, 65, 16] is both diverse and contains useful interaction in scenes similar to those we would like robots to interact in.
Revised: Alternatively, data of humans interacting in the world [14, 65, 16] is both diverse and contains useful interaction in scenes similar to those we would like robots to interact in.

Original: Of the many human video datasets, we leverage the Ego4D dataset [16] due to it’s diversity and size, although in principle our method can be used on any suitable video dataset.
Revised: Of the many human video datasets, we leverage the Ego4D dataset [16] due to it’s diversity and size, although in principle our method can be used on any suitable video dataset.

Original: Ego4D contains videos of people engaging in a wide range of tasks from cooking to socializing to assembling objects from more than 70 locations across the globe, and in total contains more than 3500 hours of data.
Revised: Ego4D contains videos of people engaging in a wide range of tasks from cooking to socializing to assembling objects from more than 70 locations across the globe, and in total contains more than 3500 hours of data.

Original: Each video clip also contains a natural language annotation describing the behavior of the person in the video (See Figure 2 (left)).
Revised: Each video clip also contains a natural language annotation describing the behavior of the person in the video (See Figure 2 (left)).

Original: 3.3 Training R3M
Revised: 3.3 Training R3M

Original: What should a good representation for robotic manipulation from human video data capture?
Revised: What should a good representation for robotic manipulation from human video data capture?

Original: We propose three key components: (1) it should capture temporal dynamics, as the agent will be sequentially interacting in the environment to accomplish tasks, (2) it should capture semantically relevant features, and (3) it should be compact.
Revised: We propose three key components: (1) it should capture temporal dynamics, as the agent will be sequentially interacting in the environment to accomplish tasks, (2) it should capture semantically relevant features, and (3) it should be compact.

Original: We next describe how we use time contrastive learning to capture (1), video-language alignment for (2), and the use of L1 regularization to encourage (3).
Revised: We next describe how we use time contrastive learning to capture (1), video-language alignment for (2), and the use of L1 regularization to encourage (3).

Original: See Figure 2 (right) for an overview of our training objective.
Revised: See Figure 2 (right) for an overview of our training objective.

Original: Time Contrastive Learning.
Revised: Time Contrastive Learning.

Original: To encourage F ϕ to capture features relevant to physical interaction and sequential decision making, the first part of our objective is a time contrastive loss [61].
Revised: To encourage F ϕ to capture features relevant to physical interaction and sequential decision making, the first part of our objective is a time contrastive loss [61].

Original: Given a batch of videos we train the encoder to produce a representation such that the distance between images closer in time is smaller than for images farther in time or from different videos.
Revised: Given a batch of videos we train the encoder to produce a representation such that the distance between images closer in time is smaller than for images farther in time or from different videos.

Original: Specifically, we sample a batch of sequences of frames [ I i , I j>i , I k>j ] 1: B , then minimize the InfoNCE loss [67]:
Revised: Specifically, we sample a batch of sequences of frames [ I i , I j>i , I k>j ] 1: B , then minimize the InfoNCE loss [67]:

Original: [Equation]
Revised: [Equation]

Original: where z = F ϕ ( I ) , and z ̸
Revised: where z = F ϕ ( I ) , and z ̸

Original: = b i is a negative example sampled from a different video in the batch.
Revised: = b i is a negative example sampled from a different video in the batch.

Original: S denotes a measure of similarity, which in our case is implemented as the negative L2 distance.
Revised: S denotes a measure of similarity, which in our case is implemented as the negative L2 distance.

Original: Video-Language Alignment.
Revised: Video-Language Alignment.

Original: To encourage F ϕ to capture semantically relevant features, we train a language prediction module from the embedding outputted by F ϕ .
Revised: To encourage F ϕ to capture semantically relevant features, we train a language prediction module from the embedding outputted by F ϕ .

Original: Essentially, by capturing features predictive of language, like “putting the apple on the plate”, the learned representation should capture semantically relevant parts of the scene like the plate and apple state, that are likely relevant to downstream manipulation tasks.
Revised: Essentially, by capturing features predictive of language, like “putting the apple on the plate”, the learned representation should capture semantically relevant parts of the scene like the plate and apple state, that are likely relevant to downstream manipulation tasks.

Original: Following Nair et al.
Revised: Following Nair et al.

Original: [56], we train a model G θ ( F ϕ
Revised: [56], we train a model G θ ( F ϕ

Original: ( I 0 ) ,
Revised: ( I 0 ) ,

Original: F ϕ ( I i ) , l ) that takes in an initial image I 0 , a future image I i , language l and outputs a score corresponding to if transitioning from I 0 to I i completes the language l .
Revised: F ϕ ( I i ) , l ) that takes in an initial image I 0 , a future image I i , language l and outputs a score corresponding to if transitioning from I 0 to I i completes the language l .

Original: We train the model under the objective that (1) the score should increase over the course of the video, and (2) the score should be higher for correct pairings of video/language than for incorrect pairings.
Revised: We train the model under the objective that (1) the score should increase over the course of the video, and (2) the score should be higher for correct pairings of video/language than for incorrect pairings.

Original: Again we sample a video clip and paired language [ I i , I j>i , l ] 1: B , and then train for this objective directly with a contrastive loss, that is:
Revised: Again we sample a video clip and paired language [ I i , I j>i , l ] 1: B , and then train for this objective directly with a contrastive loss, that is:

Original: [Equation]
Revised: [Equation]

Original: where again z = F ϕ ( I ) , and z ̸ = b is a negative example sampled from a different video in the batch (that does not match the language instruction l b ).
Revised: where again z = F ϕ ( I ) , and z ̸ = b is a negative example sampled from a different video in the batch (that does not match the language instruction l b ).

Original: Regularization.
Revised: Regularization.

Original: Finally, we hypothesize that sparse and compact representations benefit control, particularly in low data imitation learning.
Revised: Finally, we hypothesize that sparse and compact representations benefit control, particularly in low data imitation learning.

Original: State-distribution shift is a well studied failure mode in imitation learning [68], where policies trained with behavior cloning drift off the expert state distribution.
Revised: State-distribution shift is a well studied failure mode in imitation learning [68], where policies trained with behavior cloning drift off the expert state distribution.

Original: Reducing the effective dimensionality of the state space (which we implement with a simple L1 and L2 penalty) can help mitigate this issue, as we demonstrate in Section 4.4.
Revised: Reducing the effective dimensionality of the state space (which we implement with a simple L1 and L2 penalty) can help mitigate this issue, as we demonstrate in Section 4.4.

Original: R3M Summary & Implementation.
Revised: R3M Summary & Implementation.

Original: The final objective for training R3M is the weighted sum:
Revised: The final objective for training R3M is the weighted sum:

Original: [Equation]
Revised: [Equation]

Original: In principle, R3M can be implemented on top of any encoding architecture for F ϕ .
Revised: In principle, R3M can be implemented on top of any encoding architecture for F ϕ .

Original: In our experiments we focus on the ResNet50 architecture, and we release pre-trained R3M models with ResNet18,
Revised: In our experiments we focus on the ResNet50 architecture, and we release pre-trained R3M models with ResNet18,

Original: ResNet34, and ResNet50 architectures [69], as well as the accompanying training code.
Revised: ResNet34, and ResNet50 architectures [69], as well as the accompanying training code.

Original: During training, ϕ and θ are trained with an Adam optimizer to minimize Equation 3.
Revised: During training, ϕ and θ are trained with an Adam optimizer to minimize Equation 3.

Original: Lastly, R3M also trains with random cropping, applied at the video level (that is, within a batch all frames from the same video are cropped identically).
Revised: Lastly, R3M also trains with random cropping, applied at the video level (that is, within a batch all frames from the same video are cropped identically).

Original: Please see the appendix for further implementation details.
Revised: Please see the appendix for further implementation details.

Original:  Experiments
Revised:  Experiments

Original: In our experiments, we aim to study how the pre-trained R3M representation can be re-used for multiple downstream robot learning tasks.
Revised: In our experiments, we aim to study how the pre-trained R3M representation can be re-used for multiple downstream robot learning tasks.

Original: First , we study if R3M enables more data efficient imitation learning on unseen environments and tasks compared to existing visual representations and learning from scratch.
Revised: First , we study if R3M enables more data efficient imitation learning on unseen environments and tasks compared to existing visual representations and learning from scratch.

Original: Second , again in the data efficient imitation learning setting, we ablate the different components of the R3M training objective and observe that all components are important for final performance.
Revised: Second , again in the data efficient imitation learning setting, we ablate the different components of the R3M training objective and observe that all components are important for final performance.

Original: Third , we study if R3M can enable efficient real robot learning in a visually rich household setting.
Revised: Third , we study if R3M can enable efficient real robot learning in a visually rich household setting.

Original: Finally , in the appendix, we take a deeper look at task performance of R3M and prior methods with different amounts of data, different camera viewpoints, and different tasks.
Revised: Finally , in the appendix, we take a deeper look at task performance of R3M and prior methods with different amounts of data, different camera viewpoints, and different tasks.

Original: 4.1 Imitation Learning Evaluation Framework
Revised: 4.1 Imitation Learning Evaluation Framework

Original: Our evaluation methodology is loosely inspired by Parisi et al.
Revised: Our evaluation methodology is loosely inspired by Parisi et al.

Original: We focus on evaluating visual representations as frozen perception modules for downstream policy learning with behavior cloning.
Revised: We focus on evaluating visual representations as frozen perception modules for downstream policy learning with behavior cloning.

Original: Given a pretrained visual representation F ϕ , we form the state representation as a concatenation of the visual embedding z t = F ϕ
Revised: Given a pretrained visual representation F ϕ , we form the state representation as a concatenation of the visual embedding z t = F ϕ

Original: ( I t ) and the robot proprioceptive (e.g. joint positions and velocities) reading p t .
Revised: ( I t ) and the robot proprioceptive (e.g. joint positions and velocities) reading p t .

Original: The policy, π , is trained with a standard behavior cloning loss || a t − π ([ z t , p t ])
Revised: The policy, π , is trained with a standard behavior cloning loss || a t − π ([ z t , p t ])

Original: ||
Revised: ||

Original: We parameterize π as a two-layer MLP preceded by a BatchNorm at the input.
Revised: We parameterize π as a two-layer MLP preceded by a BatchNorm at the input.

Original: We train the agent for 20,000 steps, evaluate it online in the environment every 1000 steps, and report the best success rate achieved.
Revised: We train the agent for 20,000 steps, evaluate it online in the environment every 1000 steps, and report the best success rate achieved.

Original: For each visual representation and each task, we run 3 seeds of behavior cloning.
Revised: For each visual representation and each task, we run 3 seeds of behavior cloning.

Original: The final success rate reported on a task is the average over multiple seeds, viewpoints, and demo dataset sizes.
Revised: The final success rate reported on a task is the average over multiple seeds, viewpoints, and demo dataset sizes.

Original: Comparisons and Baselines.
Revised: Comparisons and Baselines.

Original: We compare our R3M model to three existing visual representations that have been shown to be effective for control: CLIP [12] which trains image representations to be aligned with paired natural language through contrastive learning and has been shown to be
Revised: We compare our R3M model to three existing visual representations that have been shown to be effective for control: CLIP [12] which trains image representations to be aligned with paired natural language through contrastive learning and has been shown to be

Original: [Figure]
Revised: [Figure]

Original: View
Revised: View

Original: [Figure]
Revised: [Figure]

Original: View View
Revised: View View

Original: [Figure]
Revised: [Figure]

Original: View View
Revised: View View

Original: View
Revised: View

Original: View useful for some manipulation [36] and navigation tasks [37], ImNet Supervised which uses features pre-trained for ImageNet classification task [2] and has been shown to be effective for reinforcement learning [38], and MoCo (345) (PVR) [23], which compresses and fuses the third, fourth, and fifth convolutional layers of a ResNet-50 model trained with MoCo [24] on ImageNet, and has been shown to be effective for imitation learning [23].
Revised: View useful for some manipulation [36] and navigation tasks [37], ImNet Supervised which uses features pre-trained for ImageNet classification task [2] and has been shown to be effective for reinforcement learning [38], and MoCo (345) (PVR) [23], which compresses and fuses the third, fourth, and fifth convolutional layers of a ResNet-50 model trained with MoCo [24] on ImageNet, and has been shown to be effective for imitation learning [23].

Original: We note here that our usage of the Moco (345) model differs from the setup in Parisi et al.
Revised: We note here that our usage of the Moco (345) model differs from the setup in Parisi et al.

Original: [23] in aspects like propreoception features, frame stacking etc.
Revised: [23] in aspects like propreoception features, frame stacking etc.

Original: As a result, the numerical results are not directly comparable across the two works.
Revised: As a result, the numerical results are not directly comparable across the two works.

Original: At the same time, we emphasize that all visual representations are used in the same way within our evaluation protocol.
Revised: At the same time, we emphasize that all visual representations are used in the same way within our evaluation protocol.

Original: 4.2 Simulation Environments
Revised: 4.2 Simulation Environments

Original: Next, we describe the environments and tasks used in our evaluations.
Revised: Next, we describe the environments and tasks used in our evaluations.

Original: For a comprehensive evaluation, we use three robot manipulation domains: MetaWorld [22], the Franka Kitchen environment [21], and Adroit [20] (See Figure 3).
Revised: For a comprehensive evaluation, we use three robot manipulation domains: MetaWorld [22], the Franka Kitchen environment [21], and Adroit [20] (See Figure 3).

Original: Note these environments are only used for downstream learning, and these environments and tasks are never seen during R3M training .
Revised: Note these environments are only used for downstream learning, and these environments and tasks are never seen during R3M training .

Original: In the MetaWorld environment we consider the tasks of assembling a ring onto a peg, picking and placing a block between bins, pushing a button, opening a drawer, and hammering a nail.
Revised: In the MetaWorld environment we consider the tasks of assembling a ring onto a peg, picking and placing a block between bins, pushing a button, opening a drawer, and hammering a nail.

Original: In Franka Kitchen, we learn the tasks of sliding the right door open, opening the left door, turning on the light, turning the stove top knob, and opening the microwave.
Revised: In Franka Kitchen, we learn the tasks of sliding the right door open, opening the left door, turning on the light, turning the stove top knob, and opening the microwave.

Original: Finally, in Adroit we consider the tasks of reorienting the pen to the specified position, and picking and moving the ball to specified position.
Revised: Finally, in Adroit we consider the tasks of reorienting the pen to the specified position, and picking and moving the ball to specified position.

Original: In all tasks, the agent is provided with image observations, as well as proprioceptive data of the robot (end-effector pose, joint positions, etc.)
Revised: In all tasks, the agent is provided with image observations, as well as proprioceptive data of the robot (end-effector pose, joint positions, etc.)

Original: that is concatenated to the encoded image.
Revised: that is concatenated to the encoded image.

Original: All tasks involve variation, either by varying the position of the target object in MetaWorld, the positioning of the desk in Franka
Revised: All tasks involve variation, either by varying the position of the target object in MetaWorld, the positioning of the desk in Franka

Original: Kitchen, or the chosen goals in Adroit.
Revised: Kitchen, or the chosen goals in Adroit.

Original: For a robust evaluation, we consider multiple views for each environment (See Figure 3), and 3 dataset sizes: [5 , 10 , 25] in MetaWorld and Franka Kitchen, and [25 , 50 , 100] in the more challenging Adroit environments.
Revised: For a robust evaluation, we consider multiple views for each environment (See Figure 3), and 3 dataset sizes: [5 , 10 , 25] in MetaWorld and Franka Kitchen, and [25 , 50 , 100] in the more challenging Adroit environments.

Original: Our comparisons measure performance for each environment and task, averaged over view, dataset size, and object or goal positions.
Revised: Our comparisons measure performance for each environment and task, averaged over view, dataset size, and object or goal positions.

Original: 4.3 Exp. 1: Does R3M enable efficient imitation on unseen environments and tasks?
Revised: 4.3 Exp. 1: Does R3M enable efficient imitation on unseen environments and tasks?

Original: In this first experiment, we measure the success rate of downstream imitation learning using different visual representations.
Revised: In this first experiment, we measure the success rate of downstream imitation learning using different visual representations.

Original: In Figure 4, we first notice that R3M is overall able to learn these vision based manipulation tasks in an extremely low data regime with ≈ 62% success rate, despite never seeing any data from the target environments in training the representation, while outperforming learning from scratch by more than 20%.
Revised: In Figure 4, we first notice that R3M is overall able to learn these vision based manipulation tasks in an extremely low data regime with ≈ 62% success rate, despite never seeing any data from the target environments in training the representation, while outperforming learning from scratch by more than 20%.

Original: Moreover, we observe that R3M outperforms all prior representations by more than 10% on average across all 12 tasks.
Revised: Moreover, we observe that R3M outperforms all prior representations by more than 10% on average across all 12 tasks.

Original: By training on diverse interactive video data, and with objectives that capture temporal structure and language relevance, R3M is the best performing method in all 3 environments, and on 11/12 of the tasks (See appendix for performance breakdown by task).
Revised: By training on diverse interactive video data, and with objectives that capture temporal structure and language relevance, R3M is the best performing method in all 3 environments, and on 11/12 of the tasks (See appendix for performance breakdown by task).

Original: The best two performing comparisons are CLIP and MoCo (345) (PVR), with CLIP performing better on MetaWorld, and MoCo (345) (PVR) performing better on Franka Kitchen and Adroit.
Revised: The best two performing comparisons are CLIP and MoCo (345) (PVR), with CLIP performing better on MetaWorld, and MoCo (345) (PVR) performing better on Franka Kitchen and Adroit.

Original: Unsurprisingly, learning from scratch performs poorly in the low-data regime we study.
Revised: Unsurprisingly, learning from scratch performs poorly in the low-data regime we study.

Original: Ultimately, we conclude that pre-trained visual representations are essential to good performance in the low-data
Revised: Ultimately, we conclude that pre-trained visual representations are essential to good performance in the low-data

Original: [Figure]
Revised: [Figure]

Original: imitation learning regime, and using R3M with diverse human video data is especially effective for learning representations useful for robotic manipulation.
Revised: imitation learning regime, and using R3M with diverse human video data is especially effective for learning representations useful for robotic manipulation.

Original: 4.4 Exp. 2: Which components of R3M are important?
Revised: 4.4 Exp. 2: Which components of R3M are important?

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: In this experiment, we seek to understand the different components of R3M, beginning with the objective.
Revised: In this experiment, we seek to understand the different components of R3M, beginning with the objective.

Original: Specifically, we compare the full R3M with
Revised: Specifically, we compare the full R3M with

Original: R3M(-Aug) , which does not use crop augmentations,
Revised: R3M(-Aug) , which does not use crop augmentations,

Original: R3M(-L1) , which does not include L 1 regularization, and R3M(-Lang) , which does not include include the video-language alignment loss.
Revised: R3M(-L1) , which does not include L 1 regularization, and R3M(-Lang) , which does not include include the video-language alignment loss.

Original: In Table 1, we report success rates per environment and averaged over all environments.
Revised: In Table 1, we report success rates per environment and averaged over all environments.

Original: First, we notice that on average across the three environments, we see a drop in performance of ≈ 2% from removing crop augmentation or from removing the L 1 regularization.
Revised: First, we notice that on average across the three environments, we see a drop in performance of ≈ 2% from removing crop augmentation or from removing the L 1 regularization.

Original: Interestingly, the impact of removing the sparsity regularization depends on the environment.
Revised: Interestingly, the impact of removing the sparsity regularization depends on the environment.

Original: In
Revised: In

Original: Franka Kitchen and MetaWorld, sparsity is helpful, while in Adroit removing sparsity actually helps performance slightly.
Revised: Franka Kitchen and MetaWorld, sparsity is helpful, while in Adroit removing sparsity actually helps performance slightly.

Original: We suspect this is partly due to the Adroit environment using more demonstrations, mitigating the state distribution shift issue.
Revised: We suspect this is partly due to the Adroit environment using more demonstrations, mitigating the state distribution shift issue.

Original: We see that across all environments, removing video-language alignment loss has the largest negative impact on performance, particularly in the Adroit environment.
Revised: We see that across all environments, removing video-language alignment loss has the largest negative impact on performance, particularly in the Adroit environment.

Original: We hypothesize that language alignment plays an important role in better capturing semantic features that might be predictive of objects and useful for object manipulation.
Revised: We hypothesize that language alignment plays an important role in better capturing semantic features that might be predictive of objects and useful for object manipulation.

Original: Nevertheless, we note that even in the fully self-supervised regime, our R3M model still outperforms prior state of the art visual representations like ImageNet trained MoCo (345) (PVR) [23] and CLIP [12] by a significant margin.
Revised: Nevertheless, we note that even in the fully self-supervised regime, our R3M model still outperforms prior state of the art visual representations like ImageNet trained MoCo (345) (PVR) [23] and CLIP [12] by a significant margin.

Original: [Table]
Revised: [Table]

Original: Next, we seek to answer the question: How important is the data?
Revised: Next, we seek to answer the question: How important is the data?

Original: To do so we include comparisons that disentangles the role of the dataset and the training objective.
Revised: To do so we include comparisons that disentangles the role of the dataset and the training objective.

Original: In particular, we have trained a MoCo model on the exact same frames of the Ego4D dataset used to train our
Revised: In particular, we have trained a MoCo model on the exact same frames of the Ego4D dataset used to train our

Original: R3M model (See Table 2).
Revised: R3M model (See Table 2).

Original: Additionally we compare to the MVP model [70], which trains a masked auto-encoder on Ego4D and other human video datasets.
Revised: Additionally we compare to the MVP model [70], which trains a masked auto-encoder on Ego4D and other human video datasets.

Original: We evaluate these comparisons on the Franka Kitchen and Adroit environments, and find that the MoCo-Ego4D model, which uses the same data and compute as R3M, gets an average success rate ∼ 10 % lower than R3M in both environments .
Revised: We evaluate these comparisons on the Franka Kitchen and Adroit environments, and find that the MoCo-Ego4D model, which uses the same data and compute as R3M, gets an average success rate ∼ 10 % lower than R3M in both environments .

Original: Moreover, we find the MVP models performs ∼ 20 % worse than R3M.
Revised: Moreover, we find the MVP models performs ∼ 20 % worse than R3M.

Original: This suggests that while there is indeed a large benefit coming from diverse human
Revised: This suggests that while there is indeed a large benefit coming from diverse human

Original: [Figure]
Revised: [Figure]

Original: video data compared to static ImageNet images (34% → 42% on Franka), the data is not the only source of performance improvement, and the R3M objective is particularly effective at learning a representation from such human videos, providing an additional ∼ 10 % boost in success rate.
Revised: video data compared to static ImageNet images (34% → 42%), the data is not the only source of performance improvement, and the R3M objective is particularly effective at learning a representation from such human videos, providing an additional ∼ 10 % boost in success rate.

Original: 5 Exp : Does R3M enable data efficient learning in real world environments?
Revised: 4.5 Exp. 3: Does R3M enable data efficient learning in real world environments?

Original: Finally, we test if R3M can enable data-efficient robot learning in cluttered real-world environments.
Revised: Finally, we test if R3M can enable data-efficient robot learning in cluttered real-world environments.

Original: To do so, we bring a Franka Emika Panda robot into a real graduate student apartment, and aim to learn household tasks from pixels with just 20 demonstrations per task, using the pre-trained
Revised: To do so, we bring a Franka Emika Panda robot into a real graduate student apartment, and aim to learn household tasks from pixels with just 20 demonstrations per task, using the pre-trained

Original: R3M representation.
Revised: R3M representation.

Original: We have the robot complete five tasks: (1) closing a dresser drawer, (2) picking a face mask placed randomly on a desk and placing it in the dresser drawer, (3) picking up lettuce randomly placed on a cutting board and putting in a cooking pan, (4) pushing a mug to a goal location, and (5) folding a towel (See Figure 5).
Revised: We have the robot complete five tasks: (1) closing a dresser drawer, (2) picking a face mask placed randomly on a desk and placing it in the dresser drawer, (3) picking up lettuce randomly placed on a cutting board and putting in a cooking pan, (4) pushing a mug to a goal location, and (5) folding a towel (See Figure 5).

Original: Like in our simulation experiments, we collect a small number of demonstrations and do simple behavior cloning with the pre-trained representation.
Revised: Like in our simulation experiments, we collect a small number of demonstrations and do simple behavior cloning with the pre-trained representation.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In Table 3, we report the success rates comparing R3M and
Revised: In Table 3, we report the success rates comparing R3M and

Original: CLIP, one of the stronger baselines from our evaluations in simulation.
Revised: CLIP, one of the stronger baselines from our evaluations in simulation.

Original: We observe that while the two perform similarly on the easier task of closing the drawer, R3M consistently performs better on the other four tasks (See Figure 5), which require more precise visual representations, yielding nearly double the success rate on average.
Revised: We observe that while the two perform similarly on the easier task of closing the drawer, R3M consistently performs better on the other four tasks (See Figure 5), which require more precise visual representations, yielding nearly double the success rate on average.

Original:  Limitations and Future Work
Revised:  Limitations and Future Work

Original: In this work, we set out to study if pre-training visual representations on diverse human videos can enable efficient learning of downstream robotic manipulation tasks.
Revised: In this work, we set out to study if pre-training visual representations on diverse human videos can enable efficient learning of downstream robotic manipulation tasks.

Original: While we were excited by strong results on a wide set of simulated and real robotic tasks, a number of important limitations remain.
Revised: While we were excited by strong results on a wide set of simulated and real robotic tasks, a number of important limitations remain.

Original: Our current evaluation is limited to imitation learning, specifically behavior cloning, with a small number of task demonstrations.
Revised: Our current evaluation is limited to imitation learning, specifically behavior cloning, with a small number of task demonstrations.

Original: While we would hope to see R3M be equally beneficial for other robotic learning settings like reinforcement learning, it could be the case that a good pretrained representation for RL is not the same as a good pre-trained representation for imitation.
Revised: While we would hope to see R3M be equally beneficial for other robotic learning settings like reinforcement learning, it could be the case that a good pretrained representation for RL is not the same as a good pre-trained representation for imitation.

Original: Studying how R3M performs in RL settings, and changes that may need to made to improve its performance is an exciting next step.
Revised: Studying how R3M performs in RL settings, and changes that may need to made to improve its performance is an exciting next step.

Original: The current R3M model also only provides a single-frame state representation.
Revised: The current R3M model also only provides a single-frame state representation.

Original: In principle, pre-training on human videos should be able to go beyond state representations (e.g. reward learning and task specification).
Revised: In principle, pre-training on human videos should be able to go beyond state representations (e.g. reward learning and task specification).

Original: Studying if R3M embeddings or the language grounding module can provide a useful reward signal is an interesting direction for future work.
Revised: Studying if R3M embeddings or the language grounding module can provide a useful reward signal is an interesting direction for future work.

Original: Acknowledgments
Revised: Acknowledgments

Original: The authors would like to thank the Ego4D team at Meta AI for assistance in using the dataset.
Revised: The authors would like to thank the Ego4D team at Meta AI for assistance in using the dataset.

Original: We’d also like to thank Karl Pertsch, Simone Parisi, Sidd Karamcheti, and numerous members of
Revised: We’d also like to thank Karl Pertsch, Simone Parisi, Sidd Karamcheti, and numerous members of

Original: Meta AI and the IRIS labs for valuable discussions.
Revised: Meta AI and the IRIS labs for valuable discussions.

Original: This work is in part supported by ONR grant
Revised: This work is in part supported by ONR grant

Original: N00014-22-1-2621.
Revised: N00014-22-1-2621.

Original: Finally, the authors would also like to thank Evan Coleman for assistance with the robot.
Revised: Finally, the authors would also like to thank Evan Coleman for assistance with the robot.

Original: A R3M Training Details A.1 Data Preprocessing
Revised: A R3M Training Details A.1 Data Preprocessing

Original: The Ego4D dataset consists of several hour long videos within a certain scene.
Revised: The Ego4D dataset consists of several hour long videos within a certain scene.

Original: Within each scene, there are many sub-clips, each with a natural language annotation.
Revised: Within each scene, there are many sub-clips, each with a natural language annotation.

Original: R3M trains with these shorter video clips paired with language annotations.
Revised: R3M trains with these shorter video clips paired with language annotations.

Original: For faster training R3M parses each video clip into frames (Resized and cropped to 224x224) and samples frames from a video clip individually.
Revised: For faster training R3M parses each video clip into frames (Resized and cropped to 224x224) and samples frames from a video clip individually.

Original: See the codebase for more details on the implementation of sampling the videos.
Revised: See the codebase for more details on the implementation of sampling the videos.

Original: A.2 Training Architecture and Hyper-Parameters
Revised: A.2 Training Architecture and Hyper-Parameters

Original: R3M can in principle be trained with any visual encoding architecture for F ϕ .
Revised: R3M can in principle be trained with any visual encoding architecture for F ϕ .

Original: We train with off the shelf ResNet18, 34, and 50 [69], as implemented by torchvision.models .
Revised: We train with off the shelf ResNet18, 34, and 50 [69], as implemented by torchvision.models .

Original: The language prediction head is implemented as an 5 layer MLP with sizes [2* E + L , 1024, 1024, 1024, 1024] and output a scalar score, where E is the output dimension of F ϕ and L is the output dimension of the DistilBERT [71] sentence encoder (768) from HuggingFace transformers .
Revised: The language prediction head is implemented as an 5 layer MLP with sizes [2* E + L , 1024, 1024, 1024, 1024] and output a scalar score, where E is the output dimension of F ϕ and L is the output dimension of the DistilBERT [71] sentence encoder (768) from HuggingFace transformers .

Original: During training of R3M, we use batch sizes of 16 video clips (where 5 frames are samples from each video clip: an initial image, final image, and sequence of 3 frames).
Revised: During training of R3M, we use batch sizes of 16 video clips (where 5 frames are samples from each video clip: an initial image, final image, and sequence of 3 frames).

Original: The initial and final frames are sampled from the first and last 20% of the video clip.
Revised: The initial and final frames are sampled from the first and last 20% of the video clip.

Original: R3M models are trained for one million steps in our experiments, and for 1.5 million steps in our released models, with a learning rate of 0.0001.
Revised: R3M models are trained for one million steps in our experiments, and for 1.5 million steps in our released models, with a learning rate of 0.0001.

Original: For the training objective in Equation 3, we use hyperparameters λ 1 = 1 , λ 2 = 1 , λ 3 = 0 .
Revised: For the training objective in Equation 3, we use hyperparameters λ 1 = 1 , λ 2 = 1 , λ 3 = 0 .

Original: 00001 , λ 4 = 0 .
Revised: 00001 , λ 4 = 0 .

Original: A.3 Additional Implementation Details
Revised: A.3 Additional Implementation Details

Original: In practice, we use more than one negative video example in training Equations 1 and 2.
Revised: In practice, we use more than one negative video example in training Equations 1 and 2.

Original: Instead we use 3 negative examples, sampled from different videos in the batch.
Revised: Instead we use 3 negative examples, sampled from different videos in the batch.

Original: Additionally in training for Equation 2, we consider the following positive pairs within a single batch element: Initial and Final Frames ( I 0 , I g ) , ( I 0 , I j>i ) , and ( I 0 , I k>j ) , with corresponding negatives ( I 0 , I 0 ) , ( I 0 , I i ) , and ( I 0 , I j ) respectively.
Revised: Additionally in training for Equation 2, we consider the following positive pairs within a single batch element: Initial and Final Frames ( I 0 , I g ) , ( I 0 , I j>i ) , and ( I 0 , I k>j ) , with corresponding negatives ( I 0 , I 0 ) , ( I 0 , I i ) , and ( I 0 , I j ) respectively.

Original: Using a larger number of positive examples from a single video and multiple negative examples from different videos stabilizes training.
Revised: Using a larger number of positive examples from a single video and multiple negative examples from different videos stabilizes training.

Original: A.4 Example Usage
Revised: A.4 Example Usage

Original: Using R3M is simple.
Revised: Using R3M is simple.

Original: The codebase is located at https://github.com/facebookresearch/r3m .
Revised: The codebase is located at https://sites.google.com/view/ robot-r3m-anon/ .

Original: Simply clone the repo and install via pip install -e .
Revised: Simply clone the repo and install via pip install -e .

Original: Then R3M can be loaded by running:from r3m import load_r3m
Revised: Then R3M can be loaded by running:

Original: B Evaluation Details B.1 Simulation Environments
Revised: B Evaluation Details B.1 Simulation Environments

Original: We focus on three simulation environments: Franka Kitchen, MetaWorld, and Adroit.
Revised: We focus on three simulation environments: Franka Kitchen, MetaWorld, and Adroit.

Original: Franka Kitchen .
Revised: Franka Kitchen .

Original: The Franka Kitchen environments used in this paper are modified from the original environment; specifically, we add additional randomization to the scene.
Revised: The Franka Kitchen environments used in this paper are modified from the original environment; specifically, we add additional randomization to the scene.

Original: We randomly change the position of the kitchen between episodes, making the task significantly more challenging both in perception and control.
Revised: We randomly change the position of the kitchen between episodes, making the task significantly more challenging both in perception and control.

Original: The 5 tasks in the Franka Kitchen involve opening the left door, opening the sliding door, turning on the light, turning the knob, and opening the microwave.
Revised: The 5 tasks in the Franka Kitchen involve opening the left door, opening the sliding door, turning on the light, turning the knob, and opening the microwave.

Original: All Franka tasks include proprioceptive data
Revised: All Franka tasks include proprioceptive data

Original: [Figure]
Revised: [Figure]

Original: of the arm joint positions and gripper positions.
Revised: of the arm joint positions and gripper positions.

Original: The horizon for all Franka tasks is 50 steps, and our imitation experiments use either 5, 10, or 25 demos.
Revised: The horizon for all Franka tasks is 50 steps, and our imitation experiments use either 5, 10, or 25 demos.

Original: MetaWorld .
Revised: MetaWorld .

Original: The MetaWorld environments are the standard V2 Button Pressing, Bin Picking, Drawer
Revised: The MetaWorld environments are the standard V2 Button Pressing, Bin Picking, Drawer

Original: Opening, Hammer, and Assembly environments available in MetaWorld [22].
Revised: Opening, Hammer, and Assembly environments available in MetaWorld [22].

Original: In all tasks, the target object (drawer, peg, block, etc.) position is randomized between episodes.
Revised: In all tasks, the target object (drawer, peg, block, etc.) position is randomized between episodes.

Original: All MetaWorld tasks include proprioceptive data of the gripper end effector pose and gripper open/- close.
Revised: All MetaWorld tasks include proprioceptive data of the gripper end effector pose and gripper open/- close.

Original: The horizon for all MetaWorld tasks is 500 steps, and our imitation experiments use either 5, 10, or 25 demos.
Revised: The horizon for all MetaWorld tasks is 500 steps, and our imitation experiments use either 5, 10, or 25 demos.

Original: Adroit .
Revised: Adroit .

Original: We use the standard Pen and Relocate tasks in the Adroit hand manipulation suite.
Revised: We use the standard Pen and Relocate tasks in the Adroit hand manipulation suite.

Original: The goal position of the pen and the goal position of the ball are randomized between episodes, and specified visually.
Revised: The goal position of the pen and the goal position of the ball are randomized between episodes, and specified visually.

Original: All Adroit tasks include proprioceptive data of the hand joints, and in the Relocate task also includes the global position of the hand.
Revised: All Adroit tasks include proprioceptive data of the hand joints, and in the Relocate task also includes the global position of the hand.

Original: The horizon for the Pen task is 100 steps and for the Relocate task is 200 steps.
Revised: The horizon for the Pen task is 100 steps and for the Relocate task is 200 steps.

Original: Our imitation experiments use either 25, 50, or 100 demos.
Revised: Our imitation experiments use either 25, 50, or 100 demos.

Original: B.2 Real World Environments
Revised: B.2 Real World Environments

Original: Our real world experiments involve bringing a Franka Emika Panda robot into a real graduate student apartment.
Revised: Our real world experiments involve bringing a Franka Emika Panda robot into a real graduate student apartment.

Original: The tasks involve putting lettuce in a pan in the kitchen, pushing a mug to a goal position on a dining table, closing a drawer, putting a mask in a drawer, and folding a towel (See Figure 6).
Revised: The tasks involve putting lettuce in a pan in the kitchen, pushing a mug to a goal position on a dining table, closing a drawer, putting a mask in a drawer, and folding a towel (See Figure 6).

Original: All tasks involve randomization (e.g. the towel/lettuce/mug/mask position or drawer position).
Revised: All tasks involve randomization (e.g. the towel/lettuce/mug/mask position or drawer position).

Original: The initial state of the gripper is also randomized each episode.
Revised: The initial state of the gripper is also randomized each episode.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The robot observation includes RGB images froma USB webcam, positioned differently for each task (See Figure 7).
Revised: The robot observation includes RGB images froma USB webcam, positioned differently for each task (See Figure 7).

Original: The robot end effector position is also concatenated with the image embedding during imitation learning.
Revised: The robot end effector position is also concatenated with the image embedding during imitation learning.

Original: B.3 Demo Data Collection
Revised: B.3 Demo Data Collection

Original: In the Franka Kitchen and Adroit tasks, expert data is generated by training a state based agent with model free RL [20].
Revised: In the Franka Kitchen and Adroit tasks, expert data is generated by training a state based agent with model free RL [20].

Original: The state based trajectories are then replayed and rendered with image observations.
Revised: The state based trajectories are then replayed and rendered with image observations.

Original: In the MetaWorld environment, a heuristic policy using state information is used to generate expert data, which is then replayed and rendered with image observations.
Revised: In the MetaWorld environment, a heuristic policy using state information is used to generate expert data, which is then replayed and rendered with image observations.

Original: On the real robot, demonstrations are collected by a human tele-operator with a PlayStation controller.
Revised: On the real robot, demonstrations are collected by a human tele-operator with a PlayStation controller.

Original: The control is applied directly in the end effector Cartesian space, and the demo trajectories are directly saved with visual observations.
Revised: The control is applied directly in the end effector Cartesian space, and the demo trajectories are directly saved with visual observations.

Original: B.4 Comparisons
Revised: B.4 Comparisons

Original: In all experiments all models use a ResNet50 base architecture.
Revised: In all experiments all models use a ResNet50 base architecture.

Original: CLIP : The CLIP comparison uses the of the shelf CLIP RN50 model available at https://github. com/openai/CLIP .
Revised: CLIP : The CLIP comparison uses the of the shelf CLIP RN50 model available at https://github. com/openai/CLIP .

Original: ImNet Supervised : This comparison uses the default ResNet architecture available from torchvision.models with pretrained=
Revised: ImNet Supervised : This comparison uses the default ResNet architecture available from torchvision.models with pretrained=

Original: True .
Revised: True .

Original: MoCo (345) :
Revised: MoCo (345) :

Original: This comparison uses a pre-trained MoCo model on Imagenet which fuses the third, fourth, and fifth convolutional layers as proposed in [23].
Revised: This comparison uses a pre-trained MoCo model on Imagenet which fuses the third, fourth, and fifth convolutional layers as proposed in [23].

Original: Note that our usage of the Moco (345) model differs from the setup in Parisi et al.
Revised: Note that our usage of the Moco (345) model differs from the setup in Parisi et al.

Original: [23] in aspects like proprioception features, frame stacking etc.
Revised: [23] in aspects like proprioception features, frame stacking etc.

Original: As a result, the numerical results are not directly comparable across the two works.
Revised: As a result, the numerical results are not directly comparable across the two works.

Original: Scratch : uses the default ResNet architecture available from torchvision.models with pretrained
Revised: Scratch : uses the default ResNet architecture available from torchvision.models with pretrained

Original: =
Revised: =

Original: False .
Revised: False .

Original: Additionally, it lets gradients from the behavior cloning MSE loss pass into the visual encoder.
Revised: Additionally, it lets gradients from the behavior cloning MSE loss pass into the visual encoder.

Original: MoCo-Ego4D : This comparison uses a pre-trained MoCo model on the samed data as R3M from the Ego4D dataset

Original: MVP :

Original: This comparison uses a pretrained MVP

Original: [40, 70] model, which trains an MAE on Ego4D and other human video datasets.

Original: B.5 Behavior Cloning Hyperparameters
Revised: B.5 Behavior Cloning Hyperparameters

Original: The downstream policy is a 2 layer MLP with hidden sizes [256,256] preceded by a BatchNorm.
Revised: The downstream policy is a 2 layer MLP with hidden sizes [256,256] preceded by a BatchNorm.

Original: The input to the policy is the concatenated visual embedding and proprioceptive data, and the output is the action.
Revised: The input to the policy is the concatenated visual embedding and proprioceptive data, and the output is the action.

Original: The policy is trained with a learning rate of 0.001, and a batch size of 32 for 20000 steps, evaluating every 1000.
Revised: The policy is trained with a learning rate of 0.001, and a batch size of 32 for 20000 steps, evaluating every 1000.

Original: [Figure] C Additional Results C.1 How does performance vary across viewpoint and demo dataset size?
Revised: C Additional Results C.1 How does performance vary across viewpoint and demo dataset size?

Original: In our next experiment, we take a closer look at R3M performance compared to prior methods across viewpoints and dataset sizes.
Revised: In our next experiment, we take a closer look at R3M performance compared to prior methods across viewpoints and dataset sizes.

Original: In Figure 8, we plot the average success rate of each method across  each dataset size and viewpoint.
Revised: In Figure 8, we plot the average success rate of each method across [Figure] each dataset size and viewpoint.

Original: We observe that the performance improvement of R3M is consistent across all viewpoints, and it is the highest performing representation in all cases.
Revised: We observe that the performance improvement of R3M is consistent across all viewpoints, and it is the highest performing representation in all cases.

Original: Interestingly, we see that the same does not hold amongst the prior methods, where the ranking between MoCo (345) and CLIP changes based on the chosen viewpoint.
Revised: Interestingly, we see that the same does not hold amongst the prior methods, where the ranking between MoCo (345) and CLIP changes based on the chosen viewpoint.

Original: Additionally, we also study the impact of dataset size for imitation learning.
Revised: Additionally, we also study the impact of dataset size for imitation learning.

Original: Again, we observe that the performance improvement from R3M is consistent, outperforming the baselines across every environment and demo dataset size.
Revised: Again, we observe that the performance improvement from R3M is consistent, outperforming the baselines across every environment and demo dataset size.

Original: We observe that in the Franka Kitchen and Adroit environments, the performance gain from R3M stays consistent with increase in dataset size, even as the absolute performance of all methods improves.
Revised: We observe that in the Franka Kitchen and Adroit environments, the performance gain from R3M stays consistent with increase in dataset size, even as the absolute performance of all methods improves.

Original: Overall, we clearly observe that the performance benefit of
Revised: Overall, we clearly observe that the performance benefit of

Original: R3M is not tied to a specific viewpoint or dataset size.
Revised: R3M is not tied to a specific viewpoint or dataset size.

Original: C.2 Performance Breakdown By Task
Revised: C.2 Performance Breakdown By Task

Original: In Figure 9 we report the success rate on each task individually.
Revised: In Figure 9 we report the success rate on each task individually.

Original: Note each success rate for each method is still the average over 3 views, 3 demo sizes, and 3 seeds.
Revised: Note each success rate for each method is still the average over 3 views, 3 demo sizes, and 3 seeds.

Original: We observe that on 11/12 tasks
Revised: We observe that on 11/12 tasks

Original: R3M is the highest performing method.
Revised: R3M is the highest performing method.

Original: [Figure]
Revised: [Figure]