Original: F ROM P LAY TO P OLICY : C ONDITIONAL B EHAVIOR G ENERATION FROM U NCURATED R OBOT D ATA
Revised: F ROM P LAY TO P OLICY : C ONDITIONAL B EHAVIOR G ENERATION FROM U NCURATED R OBOT D ATA

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While large-scale sequence modeling from offline data has led to impressive performance gains in natural language and image generation, directly translating such ideas to robotics has been challenging.
Revised: While large-scale sequence modeling from offline data has led to impressive performance gains in natural language and image generation, directly translating such ideas to robotics has been challenging.

Original: One critical reason for this is that uncurated robot demonstration data, i.e. play data, collected from non-expert human demonstrators are often noisy, diverse, and distributionally multi-modal.
Revised: One critical reason for this is that uncurated robot demonstration data, i.e. play data, collected from non-expert human demonstrators are often noisy, diverse, and distributionally multi-modal.

Original: This makes extracting useful, task-centric behaviors from such data a difficult generative modeling problem.
Revised: This makes extracting useful, task-centric behaviors from such data a difficult generative modeling problem.

Original: In this work, we present Conditional Behavior Transformers (C-BeT), a method that combines the multi-modal generation ability of Behavior
Revised: In this work, we present Conditional Behavior Transformers (C-BeT), a method that combines the multi-modal generation ability of Behavior

Original: Transformer with future-conditioned goal specification.
Revised: Transformer with future-conditioned goal specification.

Original: On a suite of simulated benchmark tasks, we find that C-BeT improves upon prior state-of-the-art work in learning from play data by an average of 45 .
Revised: On a suite of simulated benchmark tasks, we find that C-BeT improves upon prior state-of-the-art work in learning from play data by an average of 45 .

Original: Further, we demonstrate for the first time that useful task-centric behaviors can be learned on a real-world robot purely from play data without any task labels or reward information.
Revised: Further, we demonstrate for the first time that useful task-centric behaviors can be learned on a real-world robot purely from play data without any task labels or reward information.

Original: Robot videos are best viewed on our project website: https://cbet-anon.github.io .
Revised: Robot videos are best viewed on our project website: play-to-policy.github.io .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine Learning is undergoing a Cambrian explosion in large generative models for applications across vision (Ramesh et al., 2022) and language (Brown et al., 2020).
Revised: Machine Learning is undergoing a Cambrian explosion in large generative models for applications across vision (Ramesh et al., 2022) and language (Brown et al., 2020).

Original: A shared property across these models is that they are trained on large and uncurated data, often scraped from the internet.
Revised: A shared property across these models is that they are trained on large and uncurated data, often scraped from the internet.

Original: Interestingly, although these models are trained without explicit task-specific labels in a self-supervised manner, they demonstrate a preternatural ability to generalize by simply conditioning the model on desirable outputs (e.g. “prompts” in text or image generation).
Revised: Interestingly, although these models are trained without explicit task-specific labels in a self-supervised manner, they demonstrate a preternatural ability to generalize by simply conditioning the model on desirable outputs (e.g. “prompts” in text or image generation).

Original: Yet, the success of conditional generation from uncurated data has remained elusive for decision making problems, particularly in robotic behavior generation.
Revised: Yet, the success of conditional generation from uncurated data has remained elusive for decision making problems, particularly in robotic behavior generation.

Original: To address this gap in behavior generation, several works (Lynch et al., 2019; Pertsch et al., 2020b) have studied the use of generative models on play data.
Revised: To address this gap in behavior generation, several works (Lynch et al., 2019; Pertsch et al., 2020b) have studied the use of generative models on play data.

Original: Here, play data is a form of offline, uncurated data that comes from either humans or a set of expert policies interacting with the environment.
Revised: Here, play data is a form of offline, uncurated data that comes from either humans or a set of expert policies interacting with the environment.

Original: However, once trained, many of these generative models require significant amounts of additional online training with task-specific rewards (Gupta et al., 2019; Singh et al., 2020).
Revised: However, once trained, many of these generative models require significant amounts of additional online training with task-specific rewards (Gupta et al., 2019; Singh et al., 2020).

Original: In order to obtain task-specific policies without online training, a new line of approaches employ offline RL to learn goal-conditioned policies (Levine et al., 2020; Ma et al., 2022).
Revised: In order to obtain task-specific policies without online training, a new line of approaches employ offline RL to learn goal-conditioned policies (Levine et al., 2020; Ma et al., 2022).

Original: These methods often require rewards or reward functions to accompany the data, either specified during data collection or inferred through hand-crafted distance metrics, for compatibility with RL training.
Revised: These methods often require rewards or reward functions to accompany the data, either specified during data collection or inferred through hand-crafted distance metrics, for compatibility with RL training.

Original: Unfortunately, for many real-world applications, data does not readily come with rewards.
Revised: Unfortunately, for many real-world applications, data does not readily come with rewards.

Original: This prompts the question: how do we learn conditional models for behavior generation from reward-free, play data?
Revised: This prompts the question: how do we learn conditional models for behavior generation from reward-free, play data?

Original: [Table]
Revised: [Table]

Original: Initialization
Revised: Initialization

Original: Grasping oven
Revised: Grasping oven

Original: Partially opening
Revised: Partially opening

Original: Retracting
Revised: Retracting

Original: Attempt to reopen
Revised: Attempt to reopen

Original: Missed grasp
Revised: Missed grasp

Original: Goal: Open oven
Revised: Goal: Open oven

Original: Result: Failure
Revised: Result: Failure

Original: Initialization
Revised: Initialization

Original: Grasping pot
Revised: Grasping pot

Original: Moving pot
Revised: Moving pot

Original: Wedging pot in sink
Revised: Wedging pot in sink

Original: Moved pot
Revised: Moved pot

Original: Retracting
Revised: Retracting

Original: Goal: Move pot to sink
Revised: Goal: Move pot to sink

Original: Result: Success
Revised: Result: Success

Original: Initialization
Revised: Initialization

Original: Grasping microwave
Revised: Grasping microwave

Original: Missed grasp
Revised: Missed grasp

Original: Re-opening microwave
Revised: Re-opening microwave

Original: Grasping Oven
Revised: Grasping Oven

Original: Opening Oven
Revised: Opening Oven

Original: Initialization
Revised: Initialization

Original: Grasping r. knob
Revised: Grasping r. knob

Original: Turning r. knob
Revised: Turning r. knob

Original: Retracting
Revised: Retracting

Original: Grasping l. knob
Revised: Grasping l. knob

Original: Turning l. knob
Revised: Turning l. knob

Original: Goal: Turn right knob & left knob
Revised: Goal: Turn right knob & left knob

Original: Result: Success
Revised: Result: Success

Original: Goal: Open microwave & oven
Revised: Goal: Open microwave & oven

Original: Result: Success
Revised: Result: Success

Original: Grasping microwave Opening microwave
Revised: Grasping microwave Opening microwave

Original: Closing microwave
Revised: Closing microwave

Original: Grasping Pot
Revised: Grasping Pot

Original: Transporting Pot
Revised: Transporting Pot

Original: Placing Pot
Revised: Placing Pot

Original: Goal: Open microwave & move pot
Revised: Goal: Open microwave & move pot

Original: Result: Failure
Revised: Result: Failure

Original: Initialization
Revised: Initialization

Original: Turning left knob
Revised: Turning left knob

Original: Retracting
Revised: Retracting

Original: Grasping Oven
Revised: Grasping Oven

Original: Opening Oven
Revised: Opening Oven

Original: Retracting
Revised: Retracting

Original: Goal: Turn left knob & open oven
Revised: Goal: Turn left knob & open oven

Original: Result: Success
Revised: Result: Success

Original: To answer this question, we turn towards transformer-based generative models that are commonplace in text generation.
Revised: To answer this question, we turn towards transformer-based generative models that are commonplace in text generation.

Original: Here, given a prompt, models like GPT-3 (Brown et al., 2020) can generate text that coherently follow or satisfy the prompt.
Revised: Here, given a prompt, models like GPT-3 (Brown et al., 2020) can generate text that coherently follow or satisfy the prompt.

Original: However, directly applying such models to behavior generation requires overcoming two significant challenges.
Revised: However, directly applying such models to behavior generation requires overcoming two significant challenges.

Original: First, unlike the discrete tokens used in text generation, behavior generation will need models that can output continuous actions while also modeling any multi-modality present in the underlying data.
Revised: First, unlike the discrete tokens used in text generation, behavior generation will need models that can output continuous actions while also modeling any multi-modality present in the underlying data.

Original: Second, unlike textual prompts that serve as conditioning for text generation, behavior generation may not have the condition and the operand be part of the same token set, and may instead require conditioning on future outcomes.
Revised: Second, unlike textual prompts that serve as conditioning for text generation, behavior generation may not have the condition and the operand be part of the same token set, and may instead require conditioning on future outcomes.

Original: In this work, we present Conditional Behavior Transformers (C-BeT), a new model for learning conditional behaviors from offline data.
Revised: In this work, we present Conditional Behavior Transformers (C-BeT), a new model for learning conditional behaviors from offline data.

Original: To produce a distribution over continuous actions instead of discrete tokens, C-BeT augments standard text generation transformers with the action discretization introduced in Behavior Transformers (BeT)
Revised: To produce a distribution over continuous actions instead of discrete tokens, C-BeT augments standard text generation transformers with the action discretization introduced in Behavior Transformers (BeT)

Original: (Shafiullah et al., 2022).
Revised: (Shafiullah et al., 2022).

Original: Conditioning in C-BeT is done by specifying desired future states as input similar to Play-Goal Conditioned Behavior Cloning (Play-GCBC) (Lynch et al., 2019).
Revised: Conditioning in C-BeT is done by specifying desired future states as input similar to Play-Goal Conditioned Behavior Cloning (Play-GCBC) (Lynch et al., 2019).

Original: By combining these two ideas, C-BeT is able to leverage the multi-modal generation capabilities of transformer models with the future conditioning capabilities of conditional policy learning.
Revised: By combining these two ideas, C-BeT is able to leverage the multi-modal generation capabilities of transformer models with the future conditioning capabilities of conditional policy learning.

Original: Importantly, C-BeT does not require any online environment interactions during training, nor the specification of rewards or Q functions needed in offline RL.
Revised: Importantly, C-BeT does not require any online environment interactions during training, nor the specification of rewards or Q functions needed in offline RL.

Original: We experimentally evaluate C-BeT on three simulated benchmarks (visual self-driving in
Revised: We experimentally evaluate C-BeT on three simulated benchmarks (visual self-driving in

Original: CARLA (Dosovitskiy et al., 2017), multi-modal block pushing (Florence et al., 2021), and simulated kitchen (Gupta et al., 2019)), and on a real Franka robot trained with play data collected by human volunteers.
Revised: CARLA (Dosovitskiy et al., 2017), multi-modal block pushing (Florence et al., 2021), and simulated kitchen (Gupta et al., 2019)), and on a real Franka robot trained with play data collected by human volunteers.

Original: The main findings from these experiments can be summarized as:
Revised: The main findings from these experiments can be summarized as:

Original: On future-conditioned tasks, C-BeT achieves significantly higher performance compared to prior work in learning from 
Revised: On future-conditioned tasks, C-BeT achieves significantly higher performance compared to prior work in learning from play.

Original: play. C-BeT demonstrates that competent visual policies for real-world tasks can be learned from fully offline multi-modal play data (rollouts visualized in Figure 1).
Revised: C-BeT demonstrates that competent visual policies for real-world tasks can be learned from fully offline multi-modal play data (rollouts visualized in Figure 1).

Original:  B ACKGROUND AND P RELIMINARIES
Revised:  B ACKGROUND AND P RELIMINARIES

Original: Play-like data: Learning from Demonstrations (Argall et al., 2009) is one of the earliest frameworks explored for behavior learning algorithms from offline data.
Revised: Play-like data: Learning from Demonstrations (Argall et al., 2009) is one of the earliest frameworks explored for behavior learning algorithms from offline data.

Original: Typically, the datasets used in these frameworks have a built in assumption that the demonstrations are collected from an expert repeatedly demonstrating a single task in exactly the same way.
Revised: Typically, the datasets used in these frameworks have a built in assumption that the demonstrations are collected from an expert repeatedly demonstrating a single task in exactly the same way.

Original: On the contrary, play datasets violate many of such assumptions, like those of expertise of the demonstrator, and the unimodality of the task and the demonstrations.
Revised: On the contrary, play datasets violate many of such assumptions, like those of expertise of the demonstrator, and the unimodality of the task and the demonstrations.

Original: Algorithms that learn from such datasets sometimes assume that the demonstrations collected are from a rational agent with possibly some latent intent in their behavior (Lynch et al., 2019).
Revised: Algorithms that learn from such datasets sometimes assume that the demonstrations collected are from a rational agent with possibly some latent intent in their behavior (Lynch et al., 2019).

Original: Note that, unlike standard offline-RL datasets (Fu et al., 2020), play-like behavior datasets neither contain fully random behaviors, nor have rewards associated with the demonstrations.
Revised: Note that, unlike standard offline-RL datasets (Fu et al., 2020), play-like behavior datasets neither contain fully random behaviors, nor have rewards associated with the demonstrations.

Original: Behavior Transformers (BeT): BeT (Shafiullah et al., 2022) is a multi-modal behavior cloning model designed particularly for tackling play-like behavior datasets.
Revised: Behavior Transformers (BeT): BeT (Shafiullah et al., 2022) is a multi-modal behavior cloning model designed particularly for tackling play-like behavior datasets.

Original: BeT uses a GPT-like transformer architecture to model the probability distribution of action given a sequence of states π ( a t | s t − h : t ) from a given dataset.
Revised: BeT uses a GPT-like transformer architecture to model the probability distribution of action given a sequence of states π ( a t | s t − h : t ) from a given dataset.

Original: However, unlike previous behavior learning algorithms, BeT does not assume a unimodal prior for the action distribution.
Revised: However, unlike previous behavior learning algorithms, BeT does not assume a unimodal prior for the action distribution.

Original: Instead, it uses a k -means discretization to bin the actions from the demonstration set into k bins, and then uses the bins to decompose each action into a discrete and continuous component.
Revised: Instead, it uses a k -means discretization to bin the actions from the demonstration set into k bins, and then uses the bins to decompose each action into a discrete and continuous component.

Original: This support for multi-modal action distributions make BeT particularly suited for multi-modal, play-like behavior datasets where unimodal behavior cloning algorithms fail.
Revised: This support for multi-modal action distributions make BeT particularly suited for multi-modal, play-like behavior datasets where unimodal behavior cloning algorithms fail.

Original: However, vanilla BeT only supports unconditonal behavior rollouts, which means that it is not possible to choose a targeted mode of behavior during BeT policy execution.
Revised: However, vanilla BeT only supports unconditonal behavior rollouts, which means that it is not possible to choose a targeted mode of behavior during BeT policy execution.

Original: Conditional behavior learning: Generally, the problem of behavior learning for an agent is considered the task of learning a policy π : O → A mapping from the environment observations to the agent’s actions that elicit some desired behavior.
Revised: Conditional behavior learning: Generally, the problem of behavior learning for an agent is considered the task of learning a policy π : O → A mapping from the environment observations to the agent’s actions that elicit some desired behavior.

Original: Conditional behavior learning is concerned with learning a policy
Revised: Conditional behavior learning is concerned with learning a policy

Original: π : O × G → A conditioned additionally on a secondary variable g sampled from a distribution p ( g ) .
Revised: π : O × G → A conditioned additionally on a secondary variable g sampled from a distribution p ( g ) .

Original: This condition variable could be specific environment states, latents (such as one-hot vectors), or even image observations.
Revised: This condition variable could be specific environment states, latents (such as one-hot vectors), or even image observations.

Original: The success of a conditioned policy can be evaluated either through pre-specified reward functions, distance function between achieved outcome g ′ and specified outcome g , or by discounted visitation probability d π ( ·| g ) =
Revised: The success of a conditioned policy can be evaluated either through pre-specified reward functions, distance function between achieved outcome g ′ and specified outcome g , or by discounted visitation probability d π ( ·| g ) =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ (cid:80) ∞ t =

Revised: [ (cid:80) ∞ t =

Original: γ t δ ( ϕ ( o t ) = g )] if a mapping ϕ between states and achieved outcome is defined (Eysenbach et al., 2022).
Revised: γ t δ ( ϕ ( o t ) = g )] if a mapping ϕ between states and achieved outcome is defined (Eysenbach et al., 2022).

Original: Goal Conditioned Behavior Cloning (GCBC):
Revised: Goal Conditioned Behavior Cloning (GCBC):

Original: In GCBC (Lynch et al., 2019; Emmons et al., 2021), the agent is presented with a dataset of (observation, action, goal) tuples ( o, a, g ) , or sequences of such tuples, and the objective of the agent is to learn a goal-conditioned behavior policy.
Revised: In GCBC (Lynch et al., 2019; Emmons et al., 2021), the agent is presented with a dataset of (observation, action, goal) tuples ( o, a, g ) , or sequences of such tuples, and the objective of the agent is to learn a goal-conditioned behavior policy.

Original: The simplest way to achieve so is by training a policy π ( · | o, g ) that maximizes the probability of the seen data π ∗ =
Revised: The simplest way to achieve so is by training a policy π ( · | o, g ) that maximizes the probability of the seen data π ∗ =

Original: arg max π (cid:81) ( o,a,g ) P [ a ∼ π ( · | o, g )] .
Revised: arg max π (cid:81) ( o,a,g ) P [ a ∼ π ( · | o, g )] .

Original: Assuming a unimodal Gaussian distribution for π ( a | o, g ) and a model parametrized by θ , this comes down to finding the parameter θ minimizing the MSE loss, θ ∗ =
Revised: Assuming a unimodal Gaussian distribution for π ( a | o, g ) and a model parametrized by θ , this comes down to finding the parameter θ minimizing the MSE loss, θ ∗ =

Original: arg min θ (cid:80) ( o,a,g ) || a − π ( o, g ; θ ) || 2 .
Revised: arg min θ (cid:80) ( o,a,g ) || a − π ( o, g ; θ ) || 2 .

Original: To make GCBC compatible with play data that inherently does not have goal labels, goal relabeling from future states is often necessary.
Revised: To make GCBC compatible with play data that inherently does not have goal labels, goal relabeling from future states is often necessary.

Original: A common form of data augmentation in training such models, useful when G ⊂ O , is hindsight data relabeling (Andrychowicz et al., 2017), where the dataset { ( o, a, g ) } is augmented with { ( o t , a, o t ′ ) | t ′ > t } by relabeling any reached state in a future timestep as a goal state and adding it to the dataset.
Revised: A common form of data augmentation in training such models, useful when G ⊂ O , is hindsight data relabeling (Andrychowicz et al., 2017), where the dataset { ( o, a, g ) } is augmented with { ( o t , a, o t ′ ) | t ′ > t } by relabeling any reached state in a future timestep as a goal state and adding it to the dataset.

Original:  A PPROACH
Revised:  A PPROACH

Original: [Figure]
Revised: [Figure]

Original: Given a dataset { ( o, a ) } ∈ O × A of sequences of (observation, action) pairs from a play dataset, our goal is to learn a behavior generation model that is capable of handling multiple tasks and multiple ways of accomplishing each task.
Revised: Given a dataset { ( o, a ) } ∈ O × A of sequences of (observation, action) pairs from a play dataset, our goal is to learn a behavior generation model that is capable of handling multiple tasks and multiple ways of accomplishing each task.

Original: At the same time, we wish to be able to extract desired behavior from the dataset in the form of a policy through our model, or, in terms of generative models, “controllably generate” our desired behavior (see Figure 2).
Revised: At the same time, we wish to be able to extract desired behavior from the dataset in the form of a policy through our model, or, in terms of generative models, “controllably generate” our desired behavior (see Figure 2).

Original: Finally, in the process of learning this controllable, conditional generative model, we wish to minimize the amount of additional human annotation or curation required in preparing the dataset.
Revised: Finally, in the process of learning this controllable, conditional generative model, we wish to minimize the amount of additional human annotation or curation required in preparing the dataset.

Original: The method we develop to address these needs is called Conditional Behavior Transformer.
Revised: The method we develop to address these needs is called Conditional Behavior Transformer.

Original: 3. C ONDITIONAL B EHAVIOR T RANSFORMERS (C-B E T)
Revised: 3. C ONDITIONAL B EHAVIOR T RANSFORMERS (C-B E T)

Original: Conditional task formulation: First, we formulate the task of learning from a play dataset as learning a conditional behavior policy, i.e. given the current state, we need to model the distribution of actions that can lead to particular future states.
Revised: Conditional task formulation: First, we formulate the task of learning from a play dataset as learning a conditional behavior policy, i.e. given the current state, we need to model the distribution of actions that can lead to particular future states.

Original: For simplicity, our formulation can be expressed as π : O × O → D ( A ) where, given a current observation o c and a future observation o g , our policy π models the distribution of the possible actions that can take the agent from o c to o g .
Revised: For simplicity, our formulation can be expressed as π : O × O → D ( A ) where, given a current observation o c and a future observation o g , our policy π models the distribution of the possible actions that can take the agent from o c to o g .

Original: Mathematically, given a set of play trajectories T , we model the distribution π ( a | o c , o g ) ≜ P τ ∈ T ( a | o c = τ t , o g = τ t ′ , t ′ > t ) .
Revised: Mathematically, given a set of play trajectories T , we model the distribution π ( a | o c , o g ) ≜ P τ ∈ T ( a | o c = τ t , o g = τ t ′ , t ′ > t ) .

Original: Next, to make our policy more robust since we operate in the partially observable setting, we replace singular observations with a sequence of observations; namely replacing o c and o g with ¯ o c
Revised: Next, to make our policy more robust since we operate in the partially observable setting, we replace singular observations with a sequence of observations; namely replacing o c and o g with ¯ o c

Original: = o (1: N ) c and ¯ o g
Revised: = o (1: N ) c and ¯ o g

Original: = o
Revised: = o

Original: (1: N ) g for some integer N .
Revised: (1: N ) g for some integer N .

Original: Thus, the final task formulation becomes learning a generative model π with:
Revised: Thus, the final task formulation becomes learning a generative model π with:

Original: [Equation]
Revised: [Equation]

Original: Architecture selection: Note that the model for our task described in the previous paragraph is necessarily multi-modal, since depending on the sequences ¯ o c and ¯ o g , there could be multiple plausible sequences of actions with non-zero probability mass.
Revised: Architecture selection: Note that the model for our task described in the previous paragraph is necessarily multi-modal, since depending on the sequences ¯ o c and ¯ o g , there could be multiple plausible sequences of actions with non-zero probability mass.

Original: As a result, we choose Behavior
Revised: As a result, we choose Behavior

Original: Transformers (BeT) (Shafiullah et al., 2022) as our generative architecture base as it can learn action generation with multiple modes.
Revised: Transformers (BeT) (Shafiullah et al., 2022) as our generative architecture base as it can learn action generation with multiple modes.

Original: We modify the input to the BeT to be a concatenation of our future conditional observation sequence and current observation sequence.
Revised: We modify the input to the BeT to be a concatenation of our future conditional observation sequence and current observation sequence.

Original: We choose to concatenate the inputs instead of stacking them, as this allows us to independently choose sequence lengths for the current and future conditional observations.
Revised: We choose to concatenate the inputs instead of stacking them, as this allows us to independently choose sequence lengths for the current and future conditional observations.

Original: Since BeT is a sequence-to-sequence model, we only consider the actions associated with the current observations as our actions.
Revised: Since BeT is a sequence-to-sequence model, we only consider the actions associated with the current observations as our actions.

Original: We show the detailed architecture of our model in Figure 3.
Revised: We show the detailed architecture of our model in Figure 3.

Original: Dataset preparation: To train a C-BeT model on our play dataset { ( o, a ) } , we will need to appropriately prepare the dataset.
Revised: Dataset preparation: To train a C-BeT model on our play dataset { ( o, a ) } , we will need to appropriately prepare the dataset.

Original: We first convert the dataset to hold sequences of observations associated with actions, { ( o t : t + N , a t : t + N ) } .
Revised: We first convert the dataset to hold sequences of observations associated with actions, { ( o t : t + N , a t : t + N ) } .

Original: Then, during training time, we dynamically augment each pair with a sequence of future observations, functionally converting our dataset into { ( o t : t + N , a t : t + N , o t ′ :
Revised: Then, during training time, we dynamically augment each pair with a sequence of future observations, functionally converting our dataset into { ( o t : t + N , a t : t + N , o t ′ :

Original: t ′ + N ′ ) } for some t ′ > t , and treat the sequence o t ′ :
Revised: t ′ + N ′ ) } for some t ′ > t , and treat the sequence o t ′ :

Original: t ′ + N ′ as ¯ o g .
Revised: t ′ + N ′ as ¯ o g .

Original: Training objective: We employ the same objective as BeT in training C-BeT. For each of the current observation and future conditional pair, we compute the BeT loss (see appendix B for details) between the ground truth actions and the predicted actions.
Revised: Training objective: We employ the same objective as BeT in training C-BeT. For each of the current observation and future conditional pair, we compute the BeT loss (see appendix B for details) between the ground truth actions and the predicted actions.

Original: We compute the focal loss (Lin et al., 2017) on the predicted action bins, and the MT-loss (Girshick, 2015) on the predicted action offsets corresponding to the action bins as described in BeT.
Revised: We compute the focal loss (Lin et al., 2017) on the predicted action bins, and the MT-loss (Girshick, 2015) on the predicted action offsets corresponding to the action bins as described in BeT.

Original: Test-time conditioning with C-BeT:
Revised: Test-time conditioning with C-BeT:

Original: During test time, we again concatenate our future conditional sequence with our current observations, and sample actions from our model according to the BeT framework.
Revised: During test time, we again concatenate our future conditional sequence with our current observations, and sample actions from our model according to the BeT framework.

Original: While in this work, we primarily condition C-BeT on future observations, we also study
Revised: While in this work, we primarily condition C-BeT on future observations, we also study

Original: [Figure]
Revised: [Figure]

Original: other ways of training and conditioning it, such as binary latent vectors denoting the modes in a trajectory in our experiments, and compare its performance to observation-conditioned C-BeT (see
Revised: other ways of training and conditioning it, such as binary latent vectors denoting the modes in a trajectory in our experiments, and compare its performance to observation-conditioned C-BeT (see

Original: Section 4.5).
Revised: Section 4.5).

Original:  C-B E T ON S IMULATED B ENCHMARKS
Revised:  C-B E T ON S IMULATED B ENCHMARKS

Original: In this section, we discuss our experiments in simulation that are designed to answer the following key questions: How well does C-BeT learn behaviors from play?
Revised: In this section, we discuss our experiments in simulation that are designed to answer the following key questions: How well does C-BeT learn behaviors from play?

Original: How important is multi-modal action modeling?
Revised: How important is multi-modal action modeling?

Original: And finally, how does C-BeT compare to other forms of conditioning?
Revised: And finally, how does C-BeT compare to other forms of conditioning?

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We compare with the following state-of-the-art methods in learning from reward-free offline data:
Revised: We compare with the following state-of-the-art methods in learning from reward-free offline data:

Original: • Goal Conditioned BC (GCBC): GCBC (Lynch et al., 2019; Emmons et al., 2021) learns a policy by optimizing the probability of seen actions given current and the end state in a trajectory.
Revised: • Goal Conditioned BC (GCBC): GCBC (Lynch et al., 2019; Emmons et al., 2021) learns a policy by optimizing the probability of seen actions given current and the end state in a trajectory.

Original: • Weighted Goal Conditioned Supervised Learning (WGCSL)
Revised: • Weighted Goal Conditioned Supervised Learning (WGCSL)

Original: (Yang et al., 2022): GCSL (Ghosh et al., 2019) is an online algorithm with multiple rounds of collecting online data, relabeling, and training a policy on that data using GCBC.
Revised: (Yang et al., 2022): GCSL (Ghosh et al., 2019) is an online algorithm with multiple rounds of collecting online data, relabeling, and training a policy on that data using GCBC.

Original: WGCSL (Yang et al., 2022) improves GCSL by learning an additional value function used to weight the GCSL loss.
Revised: WGCSL (Yang et al., 2022) improves GCSL by learning an additional value function used to weight the GCSL loss.

Original: We compare against an single-round, offline variant of WGCSL in this work.
Revised: We compare against an single-round, offline variant of WGCSL in this work.

Original: • Learning Motor Primitives from Play (Play-LMP): Play-LMP (Lynch et al., 2019) is a behavior generation algorithm that focuses on learning short ( ∼ 30 timesteps) motor primitives from play data.
Revised: • Learning Motor Primitives from Play (Play-LMP): Play-LMP (Lynch et al., 2019) is a behavior generation algorithm that focuses on learning short ( ∼ 30 timesteps) motor primitives from play data.

Original: Play-LMP does so by using a variational-autoencoder (VAE) to encode action sequences into motor program latents and decoding actions from them.
Revised: Play-LMP does so by using a variational-autoencoder (VAE) to encode action sequences into motor program latents and decoding actions from them.

Original: • Relay Imitation Learning (RIL): Relay Imitation Learning
Revised: • Relay Imitation Learning (RIL): Relay Imitation Learning

Original: (Gupta et al., 2019) is a hierarchical imitation learning with a high level controller that generates short term target state given long term goals, and a low level controller that generates action given short term target.
Revised: (Gupta et al., 2019) is a hierarchical imitation learning with a high level controller that generates short term target state given long term goals, and a low level controller that generates action given short term target.

Original: • Conditional Implicit Behavioral Cloning (C-IBC):
Revised: • Conditional Implicit Behavioral Cloning (C-IBC):

Original: Implicit behavioral cloning (Florence et al., 2021) learns an energy based model (EBM) E ( a | o ) over demos and during test samples action
Revised: Implicit behavioral cloning (Florence et al., 2021) learns an energy based model (EBM) E ( a | o ) over demos and during test samples action

Original: a given an observation o .
Revised: a given an observation o .

Original: We compare against a conditional IBC by training an EBM E ( a | o, g ) .
Revised: We compare against a conditional IBC by training an EBM E ( a | o, g ) .

Original: • Generalization Through Imitation (GTI): GTI (Mandlekar et al., 2020)
Revised: • Generalization Through Imitation (GTI): GTI (Mandlekar et al., 2020)

Original: encodes the goal condition using a CVAE, and autoregressively rolls out action sequences given observation and goal-latent.
Revised: encodes the goal condition using a CVAE, and autoregressively rolls out action sequences given observation and goal-latent.

Original: We follow their architecture and forgo collecting new trajectories with an intermediate model since that does not fit an offline framework.
Revised: We follow their architecture and forgo collecting new trajectories with an intermediate model since that does not fit an offline framework.

Original: • Offline Goal-Conditioned RL: While offline RL is generally incompatible with play data without rewards, recently some offline goal-conditioned RL algorithms achieved success by optimizing for
Revised: • Offline Goal-Conditioned RL: While offline RL is generally incompatible with play data without rewards, recently some offline goal-conditioned RL algorithms achieved success by optimizing for

Original: [Table]
Revised: [Table]

Original: a proxy reward defined through state occupancy.
Revised: a proxy reward defined through state occupancy.

Original: Our baseline, GoFAR (Ma et al., 2022), is one such algorithm that learns a goal-conditioned value function and optimizes a policy to maximize it.
Revised: Our baseline, GoFAR (Ma et al., 2022), is one such algorithm that learns a goal-conditioned value function and optimizes a policy to maximize it.

Original: • Behavior Transformers (BeT):
Revised: • Behavior Transformers (BeT):

Original: We include unconditional BeT (Sec. 2) in our baseline to understand the improvements made by the C-BeT conditioning.
Revised: We include unconditional BeT (Sec. 2) in our baseline to understand the improvements made by the C-BeT conditioning.

Original: In practice, it acts as a “random” baseline that performs the tasks without regard for the goal.
Revised: In practice, it acts as a “random” baseline that performs the tasks without regard for the goal.

Original: • Unimodal C-BeT:
Revised: • Unimodal C-BeT:

Original: We use our method without the multi-modal head introduced in BeT. This also corresponds to a variant of Decision Transformer conditioning on outcomes instead of rewards.
Revised: We use our method without the multi-modal head introduced in BeT. This also corresponds to a variant of Decision Transformer conditioning on outcomes instead of rewards.

Original: Note that neither WGCSL nor GoFAR are directly compatible with image states and goals, since they require a proxy reward function r : S × G → R .
Revised: Note that neither WGCSL nor GoFAR are directly compatible with image states and goals, since they require a proxy reward function r : S × G → R .

Original: Thus, we had to design a proxy reward function on the image representations, exp
Revised: Thus, we had to design a proxy reward function on the image representations, exp

Original: ( − ( 1 / 4 ||
Revised: ( − ( 1 / 4 ||

Original: g − s || ) 2 ) to apply them on image-based environments.
Revised: g − s || ) 2 ) to apply them on image-based environments.

Original: For a fair comparison, we also upgrade baseline Play-LMP, C-IBC, and GTI architectures by giving them sequences of observations and retrofitting them with transformers whenever applicable.
Revised: For a fair comparison, we also upgrade baseline Play-LMP, C-IBC, and GTI architectures by giving them sequences of observations and retrofitting them with transformers whenever applicable.

Original: 4. S IMULATED E NVIRONMENTS AND D ATASETS
Revised: 4. S IMULATED E NVIRONMENTS AND D ATASETS

Original: We run our algorithms and baselines on a collection of simulated environments as a benchmark to select the best algorithms to run on our real robotic setup.
Revised: We run our algorithms and baselines on a collection of simulated environments as a benchmark to select the best algorithms to run on our real robotic setup.

Original: The simulated environments are selected to cover a variety of properties that are necessary for the real world environment, such as pixel-based observations, diverse modes in the play dataset, and complex action spaces (see Figure. 4).
Revised: The simulated environments are selected to cover a variety of properties that are necessary for the real world environment, such as pixel-based observations, diverse modes in the play dataset, and complex action spaces (see Figure. 4).

Original: CARLA self-driving: CARLA (Dosovitskiy et al., 2017) is a simulated self-driving environment created using Unreal Engine.
Revised: CARLA self-driving: CARLA (Dosovitskiy et al., 2017) is a simulated self-driving environment created using Unreal Engine.

Original: In this environment, the observations are RGB pixel values of dimension (224 , 224 , 3) , and actions are two-dimensional (accelerate/brake and steer).
Revised: In this environment, the observations are RGB pixel values of dimension (224 , 224 , 3) , and actions are two-dimensional (accelerate/brake and steer).

Original: We use an environment with a fork in the road (see Figure 2) following two possible routes to the same goal, collecting 200 demonstrations in total.
Revised: We use an environment with a fork in the road (see Figure 2) following two possible routes to the same goal, collecting 200 demonstrations in total.

Original: We condition on one of the two possible routes to the goal, and at the goal where choosing either of the two modes is valid.
Revised: We condition on one of the two possible routes to the goal, and at the goal where choosing either of the two modes is valid.

Original: Multi-modal block-pushing: We use the multi-modal block-pushing environment from Florence et al.
Revised: Multi-modal block-pushing: We use the multi-modal block-pushing environment from Florence et al.

Original: (2021) for complicated multi-modal demonstrations.
Revised: (2021) for complicated multi-modal demonstrations.

Original: In this environment, an xArm robot pushes two blocks, red and green, into two square targets colored red and green.
Revised: In this environment, an xArm robot pushes two blocks, red and green, into two square targets colored red and green.

Original: All positions are randomized with some noise at episode start.
Revised: All positions are randomized with some noise at episode start.

Original: We use 1,000 demonstrations collected using a deterministic controller, and condition on just the future block positions on each baseline.
Revised: We use 1,000 demonstrations collected using a deterministic controller, and condition on just the future block positions on each baseline.

Original: Franka relay kitchen: Originally introduced in Gupta et al.
Revised: Franka relay kitchen: Originally introduced in Gupta et al.

Original: (2019), Relay Kitchen is a robotic environment in a simulated kitchen with seven possible tasks.
Revised: (2019), Relay Kitchen is a robotic environment in a simulated kitchen with seven possible tasks.

Original: A Franka Panda robot is used to manipulate the kitchen, and the associated dataset comes with 566 demonstrations collected by humans with VR controllers performing four of the seven tasks in some sequence.
Revised: A Franka Panda robot is used to manipulate the kitchen, and the associated dataset comes with 566 demonstrations collected by humans with VR controllers performing four of the seven tasks in some sequence.

Original: 4. H OW WELL DOES C-B E T LEARN BEHAVIORS FROM PLAY ?
Revised: 4. H OW WELL DOES C-B E T LEARN BEHAVIORS FROM PLAY ?

Original: On each of these environments, we train conditional behavior generation models and evaluate them on a set of conditions sampled from the dataset.
Revised: On each of these environments, we train conditional behavior generation models and evaluate them on a set of conditions sampled from the dataset.

Original: The success is defined by the model performing the same tasks as conditioned by the future outcome.
Revised: The success is defined by the model performing the same tasks as conditioned by the future outcome.

Original: We see from Table.
Revised: We see from Table.

Original: 2 that C-BeT performs significantly better compared to the baselines on all three tasks.
Revised: 2 that C-BeT performs significantly better compared to the baselines on all three tasks.

Original: BeT, as our unconditioned “random” baseline, shows the success rate of completing tasks unconditionally, and see that none of the baselines surpasses it consistently.
Revised: BeT, as our unconditioned “random” baseline, shows the success rate of completing tasks unconditionally, and see that none of the baselines surpasses it consistently.

Original: Out of the MLP-based baselines, WGCSL performs best in the state-based tasks.
Revised: Out of the MLP-based baselines, WGCSL performs best in the state-based tasks.

Original: However, GoFAR performs best on the CARLA vision based environment where the other two MLP-based baselines fail almost completely.
Revised: However, GoFAR performs best on the CARLA vision based environment where the other two MLP-based baselines fail almost completely.

Original: We note that Play-LMP performs poorly because our tasks are long-horizon and quite far from its intended motor primitive regime, which may be challenging for Play-LMP’s short-horizon auto-encoding architecture.
Revised: We note that Play-LMP performs poorly because our tasks are long-horizon and quite far from its intended motor primitive regime, which may be challenging for Play-LMP’s short-horizon auto-encoding architecture.

Original: 4. H OW IMPORTANT IS MULTI - MODAL ACTION MODELING ?
Revised: 4. H OW IMPORTANT IS MULTI - MODAL ACTION MODELING ?

Original: While we use a multi-modal behavior model in this work, it is not immediately obvious that it may be necessary.
Revised: While we use a multi-modal behavior model in this work, it is not immediately obvious that it may be necessary.

Original: Specifically, some previous outcome-conditioned policy learning works (Chen et al., 2021;
Revised: Specifically, some previous outcome-conditioned policy learning works (Chen et al., 2021;

Original: Emmons et al., 2021) implicitly assume that policies are unimodal once conditioned on an outcome.
Revised: Emmons et al., 2021) implicitly assume that policies are unimodal once conditioned on an outcome.

Original: In Table 2 the comparison between C-BeT and unimodal C-BeT shows that this assumption may not be true for all environments, and all else being equal, having an explicitly multi-modal model helps learning an outcome conditioned policy when there may be multiple ways to achieve an outcome.
Revised: In Table 2 the comparison between C-BeT and unimodal C-BeT shows that this assumption may not be true for all environments, and all else being equal, having an explicitly multi-modal model helps learning an outcome conditioned policy when there may be multiple ways to achieve an outcome.

Original: 4. H OW DOES C-B E T COMPARE TO OTHER FORMS OF CONDITIONING ?
Revised: 4. H OW DOES C-B E T COMPARE TO OTHER FORMS OF CONDITIONING ?

Original: [Table]
Revised: [Table]

Original: We consider the question of how much comparative advantage there is in getting human labels for our tasks.
Revised: We consider the question of how much comparative advantage there is in getting human labels for our tasks.

Original: We do so by adding manual one-hot (CARLA, BlockPush) or binary (Kitchen) labels to our tasks, and training and evaluating C-BeT with those labels.
Revised: We do so by adding manual one-hot (CARLA, BlockPush) or binary (Kitchen) labels to our tasks, and training and evaluating C-BeT with those labels.

Original: As we see on Table 3,while C-BeT is able to get close to peak performance without human labels ontwo out of the three  environments,  on BlockPush adding human labels significantly improves the performance.
Revised: As we see on Table 3, on the three simulated environments, C-BeT conditioned on only future observations performs comparably to conditioning with human labels.

Original:  C-B E T ON R EAL -W ORLD R OBOTIC M ANIPULATION
Revised:  C-B E T ON R EAL -W ORLD R OBOTIC M ANIPULATION

Original: We now discuss our robot experiments, which are geared towards understanding the usefulness of
Revised: We now discuss our robot experiments, which are geared towards understanding the usefulness of

Original: C-BeT on real-world play data.
Revised: C-BeT on real-world play data.

Original: 5. R OBOTIC E NVIRONMENT AND D ATASET
Revised: 5. R OBOTIC E NVIRONMENT AND D ATASET

Original: Robot setup: Our environment consists of a Franka Emika Panda robot, similar to the simulated
Revised: Robot setup: Our environment consists of a Franka Emika Panda robot, similar to the simulated

Original: Franka Kitchen environment, set up with a children’s toy kitchen set (see Figure 1).
Revised: Franka Kitchen environment, set up with a children’s toy kitchen set (see Figure 1).

Original: The toy kitchen has an oven, a microwave, a pot, and two stove knobs that are relevant to our play dataset.
Revised: The toy kitchen has an oven, a microwave, a pot, and two stove knobs that are relevant to our play dataset.

Original: The action space in this environment contains the seven joint angle deltas normalized within the [ − 1 , 1] range, and a binary gripper control.
Revised: The action space in this environment contains the seven joint angle deltas normalized within the [ − 1 , 1] range, and a binary gripper control.

Original: Play dataset: We collected 460 sequences totaling to 265 minutes (about 4.5 hours) of play data on the toy kitchen with volunteers using a Vive VR controller to move the Franka.
Revised: Play dataset: We collected 460 sequences totaling to 265 minutes (about 4.5 hours) of play data on the toy kitchen with volunteers using a Vive VR controller to move the Franka.

Original: While collecting the play data, we did not give the volunteers any explicit instructions about doing any particular tasks, or number of tasks, beyond specifying the interactable items, and stipulating that the pot only goes on the left stove or in the sink, to prevent dropping the pot and reaching an unresettable state.
Revised: While collecting the play data, we did not give the volunteers any explicit instructions about doing any particular tasks, or number of tasks, beyond specifying the interactable items, and stipulating that the pot only goes on the left stove or in the sink, to prevent dropping the pot and reaching an unresettable state.

Original: As the observations, we save the RGB observations from two cameras on the left and right of the setup, as well as the robot’s proprioceptive joint angles.
Revised: As the observations, we save the RGB observations from two cameras on the left and right of the setup, as well as the robot’s proprioceptive joint angles.

Original: Overall, the dataset contains 45 287 frames of play interactions and their associated actions.
Revised: Overall, the dataset contains 45 287 frames of play interactions and their associated actions.

Original: Representation learning To simplify the task of learning policies on image space, we decouple the task of image representation learning from policy learning following Pari et al.
Revised: Representation learning To simplify the task of learning policies on image space, we decouple the task of image representation learning from policy learning following Pari et al.

Original: For each camera, we first fine-tune a pretrained ResNet-18 (He et al., 2016) encoder on the acquired frames with BYOL self-supervision (Grill et al., 2020).
Revised: For each camera, we first fine-tune a pretrained ResNet-18 (He et al., 2016) encoder on the acquired frames with BYOL self-supervision (Grill et al., 2020).

Original: Then, during policy learning and evaluation, instead of the image from the cameras, we pass the two 512-dimensional BYOL embeddings as part of the observation.
Revised: Then, during policy learning and evaluation, instead of the image from the cameras, we pass the two 512-dimensional BYOL embeddings as part of the observation.

Original: For the proprioceptive part of the observation, we repeat the (sin , cos) of seven joint states 74 times to get a 1036-dimensional proprioceptive representation, making our overall observation representation 2060-dimensional.
Revised: For the proprioceptive part of the observation, we repeat the (sin , cos) of seven joint states 74 times to get a 1036-dimensional proprioceptive representation, making our overall observation representation 2060-dimensional.

Original: 5. C ONDITIONAL B EHAVIOR G ENERATION ON R EAL R OBOT
Revised: 5. C ONDITIONAL B EHAVIOR G ENERATION ON R EAL R OBOT

Original: Behavior generation on single tasks: Our first experiment in the real robot is about extracting single-task policies from the play dataset.
Revised: Behavior generation on single tasks: Our first experiment in the real robot is about extracting single-task policies from the play dataset.

Original: We define our tasks as manipulating the four types of interactable objects one at a time: opening the oven door, opening the microwave door, moving the pot from the stove to the sink, and rotating a knob 90 degrees to the right.
Revised: We define our tasks as manipulating the four types of interactable objects one at a time: opening the oven door, opening the microwave door, moving the pot from the stove to the sink, and rotating a knob 90 degrees to the right.

Original: We use appropriate conditioning frames from our observation dataset, and start the robot from the neutral state to complete the four tasks.
Revised: We use appropriate conditioning frames from our observation dataset, and start the robot from the neutral state to complete the four tasks.

Original: The result of this experiment is presented in Table 4.
Revised: The result of this experiment is presented in Table 4.

Original: We see that on single task conditionals,
Revised: We see that on single task conditionals,

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Unimodal C
Revised: Unimodal C-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Multimodal C- [Table] [Table] [Table]
Revised: Multimodal C-BeT (5 , 4) /(

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C-BeT is able to complete all tasks except the knobs consistently, outperforming all our baselines, showing that C-BeT is able to extract single-task policies out of uncurated, real-world play data.
Revised: C-BeT is able to complete all tasks except the knobs consistently, outperforming all our baselines, showing that C-BeT is able to extract single-task policies out of uncurated, real-world play data.

Original: We discuss failures of C-BeT on the knob tasks in Section 5.3.
Revised: We discuss failures of C-BeT on the knob tasks in Section 5.3.

Original: While our GoFAR baseline was able to move towards the task targets, it was unable to successfully grasp or interact with any of the target objects.
Revised: While our GoFAR baseline was able to move towards the task targets, it was unable to successfully grasp or interact with any of the target objects.

Original: We believe it may be the case because unlike the robot experiment in Ma et al.
Revised: We believe it may be the case because unlike the robot experiment in Ma et al.

Original: (2022), we do not have the underlying environment state, the tasks are much more complicated, and our dataset is an order of magnitude smaller ( 400 K vs 45 K).
Revised: (2022), we do not have the underlying environment state, the tasks are much more complicated, and our dataset is an order of magnitude smaller ( 400 K vs 45 K).

Original: Behavior generation for longer horizons: Next, we ask how well our models work for longerhorizon conditioning with multiple tasks.
Revised: Behavior generation for longer horizons: Next, we ask how well our models work for longerhorizon conditioning with multiple tasks.

Original: We choose play sequences from the dataset with multiple tasks completed and use their associated states as the conditions for our models.
Revised: We choose play sequences from the dataset with multiple tasks completed and use their associated states as the conditions for our models.

Original: In our roll-outs, we calculate how many tasks completed in the original sequence were also completed in the conditional roll-outs.
Revised: In our roll-outs, we calculate how many tasks completed in the original sequence were also completed in the conditional roll-outs.

Original: We calculate this metric over 3 conditioning sequences, and report the results in Table 5.
Revised: We calculate this metric over 3 conditioning sequences, and report the results in Table 5.

Original: We see that even without any high level controller, C-BeT is able to stitch together multiple tasks from play demonstrations to complete long-horizon goals.
Revised: We see that even without any high level controller, C-BeT is able to stitch together multiple tasks from play demonstrations to complete long-horizon goals.

Original: Generalization to prompt and environment perturbations: A major requirement from any robot system deployed in the real world is to generalize to novel scenarios.
Revised: Generalization to prompt and environment perturbations: A major requirement from any robot system deployed in the real world is to generalize to novel scenarios.

Original: We evaluate the generalizability of our learned policies in two different ways.
Revised: We evaluate the generalizability of our learned policies in two different ways.

Original: In the first set of experiments, we collect fresh demonstrations that were not in the training set, and we condition our policies on such trajectories.
Revised: In the first set of experiments, we collect fresh demonstrations that were not in the training set, and we condition our policies on such trajectories.

Original: We find that across the different tasks, even with unseen conditionings, C-BeT retains 67% of the single-task performance, with 16 / 50 task successes in total.
Revised: We find that across the different tasks, even with unseen conditionings, C-BeT retains 67% of the single-task performance, with 16 / 50 task successes in total.

Original: In the second set of experiments, we add environmental distractors in the setup (Figure 1, bottom three rows) and run the single- and multi-task conditions on the modified environments.
Revised: In the second set of experiments, we add environmental distractors in the setup (Figure 1, bottom three rows) and run the single- and multi-task conditions on the modified environments.

Original: We see once again that the performance drops to around 67% of original with two distractors on the scene, but if we keep adding (four or more) distractors, the robot is unable to complete any tasks.
Revised: We see once again that the performance drops to around 67% of original with two distractors on the scene, but if we keep adding (four or more) distractors, the robot is unable to complete any tasks.

Original: 5. A NALYSIS OF F AILURE M ODES
Revised: 5. A NALYSIS OF F AILURE M ODES

Original: We see a few failure modes in our experiments that may provide additional insights into learning from real-world play data.
Revised: We see a few failure modes in our experiments that may provide additional insights into learning from real-world play data.

Original: We discuss the most salient ones in this section.
Revised: We discuss the most salient ones in this section.

Original: Failure in knob operation in the real world: We see that in all of our real world experiments, the accuracy in operating the knob is consistently lower than all other tasks.
Revised: Failure in knob operation in the real world: We see that in all of our real world experiments, the accuracy in operating the knob is consistently lower than all other tasks.

Original: This is due to the failure of the learned representations.
Revised: This is due to the failure of the learned representations.

Original: Upon inspection of the dataset images’ nearest neighbors in the representation space, we see that the BYOL-trained representation cannot identify the knob state better than random chance: the returned nearest neighbor differs in knob status often.
Revised: Upon inspection of the dataset images’ nearest neighbors in the representation space, we see that the BYOL-trained representation cannot identify the knob state better than random chance: the returned nearest neighbor differs in knob status often.

Original: Since the representation cannot identify the knob status properly, conditioning on it naturally fails.
Revised: Since the representation cannot identify the knob status properly, conditioning on it naturally fails.

Original: Importance of a multi-modal policy architecture:
Revised: Importance of a multi-modal policy architecture:

Original: One of our motivations behind incorporating the BeT architecture in our work is its ability to learn multi-modal action distributions.
Revised: One of our motivations behind incorporating the BeT architecture in our work is its ability to learn multi-modal action distributions.

Original: In our experiments, we show that for some single-task conditions such as opening the oven door, having no multi-modality is sufficient (Table 4), but for more complicated tasks and learning from a more interconnected form of play data, it is always the case that a multi-modal architecture prevents our policies from collapsing to sub-optimal solutions (Table 5).
Revised: In our experiments, we show that for some single-task conditions such as opening the oven door, having no multi-modality is sufficient (Table 4), but for more complicated tasks and learning from a more interconnected form of play data, it is always the case that a multi-modal architecture prevents our policies from collapsing to sub-optimal solutions (Table 5).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Outcome-conditioned behavior learning: Behavior learning conditioned on particular outcomes, such as reward or goals, is a long studied problem (Kaelbling, 1993; Schaul et al., 2015; Veeriah et al., 2018;
Revised: Outcome-conditioned behavior learning: Behavior learning conditioned on particular outcomes, such as reward or goals, is a long studied problem (Kaelbling, 1993; Schaul et al., 2015; Veeriah et al., 2018;

Original: Zhao et al., 2019).
Revised: Zhao et al., 2019).

Original: Compared to standard behavior learning, learning conditioned behavior can generally be more demanding since the same model can be expected to learn a multitude of behaviors depending on the outcome, which can make learning long-term behavior harder (Levy et al., 2017; Nachum et al., 2018).
Revised: Compared to standard behavior learning, learning conditioned behavior can generally be more demanding since the same model can be expected to learn a multitude of behaviors depending on the outcome, which can make learning long-term behavior harder (Levy et al., 2017; Nachum et al., 2018).

Original: As a result, a common line of work in outcome-conditioned learning is to use some form of relabeling of demonstrations or experience buffer as a form of data augmentation (Kaelbling, 1993; Andrychowicz et al., 2017; Ghosh et al., 2019) similar to what we do in the paper.
Revised: As a result, a common line of work in outcome-conditioned learning is to use some form of relabeling of demonstrations or experience buffer as a form of data augmentation (Kaelbling, 1993; Andrychowicz et al., 2017; Ghosh et al., 2019; Goyal et al., 2022) similar to what we do in the paper.

Original: As opposed to goal or state conditioned learning, which we focus on in this paper, recently reward conditioned learning using a transformer (Chen et al., 2021) was introduced.
Revised: As opposed to goal or state conditioned learning, which we focus on in this paper, recently reward conditioned learning using a transformer (Chen et al., 2021) was introduced.

Original: However, later work found that it may not work as expected in all environments (Paster et al., 2022; Brandfonbrener et al., 2022) and large transformer models may not be necessary (Emmons et al., 2021) for reward conditioned learning.
Revised: However, later work found that it may not work as expected in all environments (Paster et al., 2022; Brandfonbrener et al., 2022) and large transformer models may not be necessary (Emmons et al., 2021) for reward conditioned learning.

Original: In this work, we find that using transformers is crucial, particularly when dealing with high dimensional visual observation and multi-modal actions.
Revised: In this work, we find that using transformers is crucial, particularly when dealing with high dimensional visual observation and multi-modal actions.

Original: Learning from play data: Our work is most closely related to previous works such as Lynch et al.
Revised: Learning from play data: Our work is most closely related to previous works such as Lynch et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: (2019), which also focus on learning from play demonstrations that may not be strictly optimal and uniformly curated for a single task.
Revised: (2019), which also focus on learning from play demonstrations that may not be strictly optimal and uniformly curated for a single task.

Original: Learning policies capable of multiple tasks from play data allows knowledge sharing, which is why it may be more efficient compared to learning from demonstrations directly (Zhang et al., 2018;
Revised: Learning policies capable of multiple tasks from play data allows knowledge sharing, which is why it may be more efficient compared to learning from demonstrations directly (Zhang et al., 2018;

Original: Rahmatizadeh et al., 2018; Duan et al., 2017; Pari et al.,  2021).
Revised: Rahmatizadeh et al., 2018; Duan et al., 2017; Pari et al., 2021; Young et al., 2021).

Original: Gupta et al.
Revised: Gupta et al.

Original: attempts reset-free learning with play data, but requires human annotation and instrumentation in the environment for goal labels.
Revised: attempts reset-free learning with play data, but requires human annotation and instrumentation in the environment for goal labels.

Original: Beyond learning policies from play data, learning self-supervised visual representations that are not entirely task-specific may also be a fruitful endeavor, as shown by Young et al.

Original: ; Nair et al. Generative modeling of behavior: Our method of learning a generative model for behavior learning follows a long line of work, including Inverse Reinforcement Learning or IRL (Russell, 1998; Ng et al., 2000; Ho & Ermon, 2016), where given expert demonstrations, a model tries to construct the reward function, which is then used to generate desirable behavior.
Revised: Generative modeling of behavior: Our method of learning a generative model for behavior learning follows a long line of work, including Inverse Reinforcement Learning or IRL (Russell, 1998; Ng et al., 2000; Ho & Ermon, 2016), where given expert demonstrations, a model tries to construct the reward function, which is then used to generate desirable behavior.

Original: Another class of algorithms learn a generative action decoder (Pertsch et al., 2020a; Singh et al., 2020) from interaction data to make downstream reinforcement learning faster and easier, nominally making multi-modal action distribution easier.
Revised: Another class of algorithms learn a generative action decoder (Pertsch et al., 2020a; Singh et al., 2020) from interaction data to make downstream reinforcement learning faster and easier, nominally making multi-modal action distribution easier.

Original: Finally, a class of algorithms, most notably Liu et al.
Revised: Finally, a class of algorithms, most notably Liu et al.

Original: Florence et al.
Revised: Florence et al.

Original: ; Kostrikov et al.
Revised: ; Kostrikov et al.

Original: (2021); Nachum & Yang (2021) do not directly learn a generative model, but instead learn energy based models that need to be sampled to generate behavior, although they do not primarily focus on goal-conditioning.
Revised: (2021); Nachum & Yang (2021) do not directly learn a generative model, but instead learn energy based models that need to be sampled to generate behavior, although they do not primarily focus on goal-conditioning.

Original: Transformers for behavior learning: Our work follows earlier notable works in using transformers to learn a behavior model from an offline dataset, such as Chen et al.
Revised: Transformers for behavior learning: Our work follows earlier notable works in using transformers to learn a behavior model from an offline dataset, such as Chen et al.

Original: ; Janner et al.
Revised: ; Janner et al.

Original: Shafiullah et al.
Revised: Shafiullah et al.

Original: Our work is most closely related to Shafiullah et al.
Revised: Our work is most closely related to Shafiullah et al.

Original: (2022) as we build on their transformer architecture, while our unimodal baseline is a variant of Chen et al.
Revised: (2022) as we build on their transformer architecture, while our unimodal baseline is a variant of Chen et al.

Original: (2021) that learns outcome conditioned instead of reward conditioned policy.
Revised: (2021) that learns outcome conditioned instead of reward conditioned policy.

Original: Beyond these, Dasari & Gupta (2020); Mandi et al.
Revised: Beyond these, Dasari & Gupta (2020); Mandi et al.

Original: (2021) summarizes historical visual context using transformers, and Clever et al.
Revised: (2021) summarizes historical visual context using transformers, and Clever et al.

Original: (2021) relies on the long-term extrapolation abilities of transformers as sequence models.
Revised: (2021) relies on the long-term extrapolation abilities of transformers as sequence models.

Original: The goal of
Revised: The goal of

Original: C-BeT is orthogonal to these use cases, but can be combined with them for future applications.
Revised: C-BeT is orthogonal to these use cases, but can be combined with them for future applications.

Original:  D ISCUSSION AND L IMITATIONS
Revised:  D ISCUSSION AND L IMITATIONS

Original: In this work, we have presented C-BeT, a new approach for conditional behavior generation that can learn from offline play data.
Revised: In this work, we have presented C-BeT, a new approach for conditional behavior generation that can learn from offline play data.

Original: Across a variety of benchmarks, both simulated and real, we find that C-BeT significantly improves upon prior state-of-the-art work.
Revised: Across a variety of benchmarks, both simulated and real, we find that C-BeT significantly improves upon prior state-of-the-art work.

Original: However, we have noticed two limitations in C-BeT, particularly for real-robot behavior learning.
Revised: However, we have noticed two limitations in C-BeT, particularly for real-robot behavior learning.

Original: First, if the features provided to
Revised: First, if the features provided to

Original: C-BeT do not appropriately capture relevant objects in the scene, the robot execution often fails to interact with that object in its environment.
Revised: C-BeT do not appropriately capture relevant objects in the scene, the robot execution often fails to interact with that object in its environment.

Original: Second, some tasks, like opening the oven door, have simpler underlying data that is not multimodal, which renders only meager gains with C-BeT.
Revised: Second, some tasks, like opening the oven door, have simpler underlying data that is not multimodal, which renders only meager gains with C-BeT.

Original: A more detailed analysis of these limitations are presented in Section 5.3.
Revised: A more detailed analysis of these limitations are presented in Section 5.3.

Original: We believe that future work in visual representation learning can address poor environment features, while the collection of even larger play datasets will provide more realistic offline data for large-scale behavior learning models.
Revised: We believe that future work in visual representation learning can address poor environment features, while the collection of even larger play datasets will provide more realistic offline data for large-scale behavior learning models.

Original: R EPRODUCIBILITY S TATEMENT
Revised: A CKNOWLEDGEMENT

Original: As a part of our commitment to reproducibility we have release our code and roll-out videos on the project website, https://cbet-anon.github.io and plan to add our dataset there.
Revised: We thank Sridhar Arunachalam, David Brandfonbrener, Irmak Guzey, Yixin Lin, Jyo Pari, Abitha Thankaraj and Austin Wang for their valuable feedback and discussions.

Original: We have already referred to the codebases that we build upon in Section C, and included the hyperparameters used to train the models in our paper. Finally, our simulated experiments are built on free environments and thus our experiments should be easily replicable
Revised: This work was supported by awards from Honda, Meta, Hyundai, Amazon and ONR awards N00014-21-1-2758 and N0001422-1-

Original: B B EHAVIOR T RANSFORMERS
Revised: B B EHAVIOR T RANSFORMERS

Original: We use Behavior Transformers from Shafiullah et al.
Revised: We use Behavior Transformers from Shafiullah et al.

Original: as our backbone architecture, building our conditional algorithm on top of it.
Revised: as our backbone architecture, building our conditional algorithm on top of it.

Original: In this section, we describe the BeT architecture and the training objective to help the readers understand the details of our algorithm.
Revised: In this section, we describe the BeT architecture and the training objective to help the readers understand the details of our algorithm.

Original: B. B E T ARCHITECTURE
Revised: B. B E T ARCHITECTURE

Original: BeT uses a repurposed MinGPT architecture to model multi-modal behavior.
Revised: BeT uses a repurposed MinGPT architecture to model multi-modal behavior.

Original: It uses the MinGPT trunk as a sequence-to-sequence model that tries to predict a sequence of actions a t : t + h given a sequence of states or observations
Revised: It uses the MinGPT trunk as a sequence-to-sequence model that tries to predict a sequence of actions a t : t + h given a sequence of states or observations

Original: o t : t + h .
Revised: o t : t + h .

Original: Beyond just prediction the actions, however, BeT tries to model the multi-modal action distribtuion given the observations.
Revised: Beyond just prediction the actions, however, BeT tries to model the multi-modal action distribtuion given the observations.

Original: To create a multi-modal model over the continuous action distribution, BeT uses an action encoder-decoder architecture that can encode each action vector into a discrete latent and a smaller-norm continuous offset.
Revised: To create a multi-modal model over the continuous action distribution, BeT uses an action encoder-decoder architecture that can encode each action vector into a discrete latent and a smaller-norm continuous offset.

Original: BeT does so by using an offline action dataset to create a k -means model of the actions.
Revised: BeT does so by using an offline action dataset to create a k -means model of the actions.

Original: Then, an action is encoded into its associated bin out of the k -bins (binning), and a small continuous offset from the associated bin.
Revised: Then, an action is encoded into its associated bin out of the k -bins (binning), and a small continuous offset from the associated bin.

Original: The BeT model, given a sequence of observations o t : t + h , predicts a k -dimensional multinomial distribution over the k -bins, as well as a k × | A | dimensional matrix for offsets associated with each action bins.
Revised: The BeT model, given a sequence of observations o t : t + h , predicts a k -dimensional multinomial distribution over the k -bins, as well as a k × | A | dimensional matrix for offsets associated with each action bins.

Original: Sampling from the BeT action distribution is done via sampling a discrete bin first, taking its associated action offset, and then adding the bin center with the action offset.
Revised: Sampling from the BeT action distribution is done via sampling a discrete bin first, taking its associated action offset, and then adding the bin center with the action offset.

Original: B. B E T TRAINING OBJECTIVE
Revised: B. B E T TRAINING OBJECTIVE

Original: Given an observation o and its associated ground truth action a , we will now present the simplified version of how the BeT loss is calculated.
Revised: Given an observation o and its associated ground truth action a , we will now present the simplified version of how the BeT loss is calculated.

Original: Let us assume the BeT model prediction is π ( o ) d ∈ R k , π ( o )
Revised: Let us assume the BeT model prediction is π ( o ) d ∈ R k , π ( o )

Original: c ∈ R k ×| A | for the discrete and the continuous parts of the predictions.
Revised: c ∈ R k ×| A | for the discrete and the continuous parts of the predictions.

Original: Let us also assume that ⌊ a ⌋ is the discrete bin out of the k bins that a belongs to, and ⟨ a ⟩ = a − BinCenter ( ⌊ a ⌋ ) .
Revised: Let us also assume that ⌊ a ⌋ is the discrete bin out of the k bins that a belongs to, and ⟨ a ⟩ = a − BinCenter ( ⌊ a ⌋ ) .

Original: Then, the BeT loss becomes
Revised: Then, the BeT loss becomes

Original: [Equation]
Revised: [Equation]

Original: Where L focal is the Focal loss (Lin et al., 2017), a special case of the negative log likelihood loss defined as
Revised: Where L focal is the Focal loss (Lin et al., 2017), a special case of the negative log likelihood loss defined as

Original: [Equation]
Revised: [Equation]

Original: and L MT is the multi-task loss (Girshick, 2015) defined as
Revised: and L MT is the multi-task loss (Girshick, 2015) defined as

Original: MT-Loss
Revised: MT-Loss

Original: [Equation]
Revised: [Equation]

Original: C I MPLEMENTATION D ETAILS C. I MPLEMENTATION USED
Revised: C I MPLEMENTATION D ETAILS C. I MPLEMENTATION USED

Original: In our work
Revised: In our work

Original: [Table]
Revised: [Table]

Original: we base our C-BeT implementation off of the official repo published at https: //github.com/notmahi/bet .
Revised: we base our C-BeT implementation off of the official repo published at https: //github.com/notmahi/bet .

Original: For the GCBC, WGCSL, and GoFAR baselines, we use the official repo released by the GoFAR authors https://github.com/JasonMa2016/GoFAR/ .
Revised: For the GCBC, WGCSL, and GoFAR baselines, we use the official repo released by the GoFAR authors https://github.com/JasonMa2016/GoFAR/ .

Original: C. H YPERPARAMETERS LIST :
Revised: C. H YPERPARAMETERS LIST :

Original: We present the C-BeT hyperparameters in Table 6 below, which were mostly using the default hyperparameters in the original Shafiullah et al.
Revised: We present the C-BeT hyperparameters in Table 6 below, which were mostly using the default hyperparameters in the original Shafiullah et al.

Original: (2022) paper:
Revised: (2022) paper:

Original: [Table]
Revised: [Table]

Original: The shared hyperparameters are in Table 7.
Revised: The shared hyperparameters are in Table 7.

Original: [Table]
Revised: [Table]

Original: D R OBOT E NVIRONMENT D EMONSTRATION T RAJECTORIES
Revised: D R OBOT E NVIRONMENT D EMONSTRATION T RAJECTORIES

Original: E
Revised: E

Original: S IMULATED E NVIRONMENT R OLLOUT T RAJECTORIES
Revised: S IMULATED E NVIRONMENT R OLLOUT T RAJECTORIES

Original: GoFAR
Revised: GoFAR

Original: WG
Revised: WG

Original: CSL
Revised: CSL

Original: GCBC
Revised: GCBC

Original: CBeT
Revised: CBeT

Original: GoFAR WGCSL GCBC
Revised: [Figure]

Original: CBeT
Revised: BeT

Original: GoFAR
Revised: GoFAR

Original: WGCS
Revised: WGCS

Original: L
Revised: L

Original: GCBC CBeT
Revised: GCBC CBeT