Original: VIOLA:Object-Centric Imitation Learning for Vision-BasedRobot Manipulation 
Revised: VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors

Original: Anonymous Author(s)

Original: AfﬁliationAddress email Abstract: We introduce VIOLA, an object-centric imitation learning approachto learning closed-loop visuomotor policies for robot manipulation.
Revised: Abstract: We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation.

Original: Our approachconstructs object-centric representations based on general object proposals from apre-trained vision model.
Revised: Our approach constructs object-centric representations based on general object proposals from a pre-trained vision model.

Original: It uses a transformer-based policy to reason over theserepresentations and attends to the task-relevant visual factors for action prediction.
Revised: VIOLA uses a transformer-based policy to reason over these representations and attend to the task-relevant visual factors for action prediction.

Original: Such object-based structural priors improve deep imitation learning algorithm’srobustness against object variations and environmental perturbations.
Revised: Such object-based structural priors improve deep imitation learning algorithm’s robustness against object variations and environmental perturbations.

Original: We quantitatively evaluate VIOLA in simulation and on real robots.
Revised: We quantitatively evaluate VIOLA in simulation and on real robots.

Original: VIOLA outperformsthe state-of-the-art imitation learning methods by 45 .
Revised: VIOLA outperforms the state-of-the-art imitation learning methods by 45 .

Original: 8% in success rates.
Revised: 8% in success rate.

Original: It hasalso been deployed successfully on a physical robot to solve challenging longhorizon tasks, such as dining table arrangements and coffee making.
Revised: It has also been deployed successfully on a physical robot to solve challenging long-horizon tasks, such as dining table arrangement and coffee making.

Original: More videosand model details can be found in supplementary materials and the anonymousproject website: https://sites.google.com/view/viola-manipulation.
Revised: More videos and model details can be found in supplementary material and the project website: https://ut-austin-rpl.github.io/VIOLA.

Original:  Introduction
Revised:  Introduction

Original: Vision-based manipulation is a critical ability for autonomous robots to interact with everyday environments.
Revised: Vision-based manipulation is a critical ability for autonomous robots to interact with everyday environments.

Original: It requires the robots to understand the unstructured world through visual perceptionto determine intelligent behaviors.
Revised: It requires the robots to understand the unstructured world through visual perception to determine intelligent behaviors.

Original: In recent years, deep imitation learning (IL)
Revised: In recent years, deep imitation learning (IL)

Original: [1–4] has emergedas a powerful approach to training visuomotor policies on diverse ofﬂine data, particularly humandemonstrations.
Revised: [1–4] has emerged as a powerful approach to training visuomotor policies on diverse ofﬂine data, particularly human demonstrations.

Original: Its successof it stems from the effectiveness of training over-parameterized neural networks end-to-end with supervised learning objectives.
Revised: Its success stems from the effectiveness of training over-parameterized neural networks end-to-end with supervised learning objectives.

Original: These models excel at mapping rawvisual observations to motor actions without manual engineering.
Revised: These models excel at mapping raw visual observations to motor actions without manual engineering.

Original: While deep IL methods oftendistinguish themselves from reinforcement learning counterparts in their scalability to long-horizontasks, a burgeoning body of recent work pointed out that IL methods lack robustness to covariateshifts and environmental perturbations [5–8].
Revised: While deep IL methods often distinguish themselves from reinforcement learning counterparts in their scalability to long-horizon tasks, a burgeoning body of recent work pointed out that IL methods lack robustness to covariate shifts and environmental perturbations [5–11].

Original: End-to-end visuomotor policies are likely to falselyassociate actions with task-irrelevant visual factors, leading to poor generalization in new situations.
Revised: End-to-end visuomotor policies are likely to falsely associate actions with task-irrelevant visual factors, leading to poor generalization in new situations.

Original: In this work, we endow imitation learning algorithms with awareness about objects and their interactions to improve their efﬁcacy and robustness in vision-based manipulation tasks.
Revised: In this work we endow imitation learning algorithms with awareness about objects and their interactions to improve their efﬁcacy and robustness in vision-based manipulation tasks.

Original: As cognitivescience studies suggest, explaining a visual scene as objects and their interactions facilitates humansto learn fast and make accurate predictions [9–11].
Revised: As cognitive science studies suggest, explaining a visual scene as objects and their interactions facilitates humans to learn fast and make accurate predictions [12–14].

Original: Inspired by these ﬁndings, we hypothesize thatdecomposing a visual scene into factorized representations of objects in the scenes would enablerobots to reason about the manipulation workspace in a modular fashion and improve their generalization ability.
Revised: Inspired by these ﬁndings, we hypothesize that decomposing a visual scene into factorized representations of objects in the scenes would enable robots to reason about the manipulation workspace in a modular fashion and improve their generalization ability.

Original: To this end, we develop an object-centric imitation learning approach, which infusesstructural object-based priors into the model architecture of visuomotor policies.
Revised: To this end, we develop an object-centric imitation learning approach, which infuses structural object-based priors into the model architecture of visuomotor policies.

Original: Training policieswith these priors would make it easier for the model to focus on task-relevant visual cues whilediscarding spurious dependencies.
Revised: Training policies with these priors would make it easier for the model to focus on task-relevant visual cues while discarding spurious dependencies.

Original: The ﬁrst and foremost challenge of such an object-centric approach is to determine what constitutes an object and how objects are represented.
Revised: The ﬁrst and foremost challenge of such an object-centric approach is to determine what constitutes an object and how objects are represented.

Original: The deﬁnitions of objects are often ﬂuid andtask-dependent for manipulation tasks.
Revised: The deﬁnitions of objects are often ﬂuid and task-dependent for manipulation tasks.

Original: This work studies the notions of objects operationally and[Figure] considers them as disentangled visual concepts that inform the robot’s decision-making.
Revised: This work studies the notions of objects operationally and considers them as disentangled visual concepts that inform the robot’s decision-making.

Revised: Previous works have explored learning visuomotor policies with awareness of objects, but they are limited to simple control domain [6], single object manipulation [15], or require costly annotations for object detection [16].

Original: We aremotivated by the recent advances in visual recognition, in particular, image models for generatingobject proposals[12, 13], i.e. , localized bounding boxes on 2D images.
Revised: We are motivated by the recent advances in visual recognition, in particular, image models for generating object proposals [17, 18], i.e. , localized bounding boxes on 2D images.

Original: These object proposals capture general priors of “objectness” across appearance variations and object categories.
Revised: These object proposals capture general priors of “objectness” across appearance variations and object cate- [Figure] gories.

Original: They haveserved as intermediate representations for downstream vision tasks, such as object detection andinstance segmentation [14–16].
Revised: They have served as intermediate representations for downstream vision tasks, such as object detection and instance segmentation [19–21].

Original: Similarly, we use object proposals from  pre-trained vision modelsas a starting point to build ourown object-centric representations.
Revised: In this work, we investigate using object proposals from a pre-trained vision models as object-centric priors for visuomotor policy in manipulation and use the object proposals as a starting point to build our object-centric representations.

Original: We introduce VIOLA ( V isuomotor I mitation via O bject-centric L e A rning), an object-centric imitation learning algorithm to train closed-loop visuomotor policies for robot manipulation.
Revised: We introduce VIOLA ( V isuomotor I mitation via O bject-centric L e A rning), an object-centric imitation learning model to train closed-loop visuomotor policies for robot manipulation.

Original: Thehigh-level overview of the method is illustrated in Figure 1.
Revised: The high-level overview of the method is illustrated in Figure 1.

Original: VIOLA ﬁrst uses a pre-trained Region 
Revised: VIOLA ﬁrst uses a pre-trained Region Proposal

Original: Proposal Network (RPN) [13] to get a set of general object proposals from raw visual observations.
Revised: Network (RPN) [18] to get a set of general object proposals from raw visual observations.

Original: We extract features from each proposal region to build the factorized object-centric representationsof the visual scene.
Revised: We extract features from each proposal region to build the factorized object-centric representations of the visual scene.

Original: These object-centric representations are converted into a set of discrete tokensand processed by a Transformer encoder [17], which is able to learn to focus on task-relevant regionswhile ignoring the irrelevant visual factors for decision-making through the multi-head self-attentionmechanism when trained on supervised imitation learning objectives.
Revised: These object-centric representations are converted into a set of discrete tokens and processed by a Transformer encoder [22]. The transformer encoder learns to focus on task-relevant regions while ignoring the irrelevant visual factors for decision-making through the multi-head selfattention mechanism when trained on supervised imitation learning objectives.

Original: We compare VIOLA against state-of-the-art deep imitation learning methods for vision-based manipulation in simulation and on a real robot.
Revised: We compare VIOLA against state-of-the-art deep imitation learning methods for vision-based manipulation in simulation and on a real robot.

Original: We use simulation to systematically evaluate the policies’ performances and generalization abilities in the canonical setting ( i.e. , testing in the trainingdistribution) and under three challenging variations, includingobject initial  placements, presence ofdistracting objects, and camera pose perturbations.
Revised: We use simulation to systematically evaluate the policies’ performances and generalization abilities in the canonical setting ( i.e. , testing in the training distribution) and under three challenging variations, including initial object placements, presence of distracting objects, and camera pose perturbations.

Original: Our quantitative evaluations show that VIOLAoutperforms the most competitive baseline by 45 .
Revised: Our quantitative evaluations show that VIOLA outperforms the most competitive baseline by 45 .

Original: 8% in success rate.
Revised: 8% in success rate.

Original: When visual variations suchas jittered camera views are introduced, VIOLA maintains its robust behaviors of precise graspingand manipulation, while end-to-end learning methods would fail to reach the target objects.
Revised: When visual variations such as jittered camera views are introduced, VIOLA maintains its robust behaviors of precise grasping and manipulation, while end-to-end learning methods would fail to reach the target objects.

Original: VIOLAalso produces visuomotor policies to solve three challenging real-world tasks with a small set of 50demonstrations, including a multi-stage coffee-making task where VIOLA achieves 60% successrate while baseline methods completely fail.
Revised: VIOLA also produces visuomotor policies to solve three challenging real-world tasks with a small set ofdemonstrations, including a multi-stage coffee-making task where VIOLA achieves 60% success rate while baseline methods fail entirely.

Original: Our contributions with VIOLA are three-fold: 1) We learn object-centric representations basedon general object proposals and design a transformer-based policy that determines task-relevantproposals to generate the robot’s actions; 2) We show that VIOLA outperforms state-of-the-artbaselines in simulation and validate the effectiveness of our model designs through ablative studies,and 3) We show that VIOLA learns policies on a real robot to complete challenging tasks.
Revised: Our contributions with VIOLA are three-fold: 1) We learn object-centric representations based on general object proposals and design a transformer-based policy that determines task-relevant proposals to generate the robot’s actions; 2) We show that VIOLA outperforms state-of-the-art baselines in simulation and validate the effectiveness of our model designs through ablative studies; and 3) We show that VIOLA learns policies on a real robot to complete challenging tasks.

Original:  Related Work
Revised:  Related Work

Original: Imitation Learning (IL) for Manipulation.
Revised: Imitation Learning (IL) for Manipulation.

Original: IL has been an established paradigm for acquiringmanipulation policies for decades.
Revised: IL has been an established paradigm for acquiring manipulation policies for decades.

Original: It can be roughly categorized into non-parametric and parametricapproaches.
Revised: It can be roughly categorized into non-parametric and parametric approaches.

Original: Non-parametric approaches, such as DMP and PrMP, can effectively acquire manipulation behaviors through a small number of demonstrations [18–21].
Revised: Non-parametric approaches, such as DMP and PrMP, can effectively acquire manipulation behaviors through a small number of demonstrations [23–26].

Original: However, they typically focuson open-loop trajectory generation and fall short in handling high-dimensional observations.
Revised: However, they typically focus on open-loop trajectory generation and fall short in handling high-dimensional observations.

Original: Parametric approaches, especially neural networks, have shown promise in vision-based manipulation.
Revised: Parametric approaches, especially neural networks, have shown promise in vision-based manipulation.

Original: Nevertheless, these approaches are susceptible to distributional shifts and observation noises [1–
Revised: Nevertheless, these approaches are susceptible to distributional shifts and observation noises [1–4, 27].

Original: VIOLA builds upon recent work on end-to-end IL and introduces object-centric priors intothe neural network policies, thus improving their robustness towards visual variations.
Revised: Object-centric priors have been previously explored in imitation learning policies to overcome the issues above [15, 16]. However, these previous works either focus on the manipulation of single object instances, or requires costly annotations for pre-training object detections. Based on the same conceptual idea as previous object-centric imitation learning, VIOLA uses a pre-trained RPN to introduce object proposals as object-centric priors into the end-to-end IL policies, thus improving their robustness towards visual variations and solving tasks that involve complicated interaction with multiple objects.

Original: Visual Representations for Visuomotor Policies.
Revised: Visual Representations for Visuomotor Policies.

Original: Various types of intermediate visual representations have been explored for visuomotor policy learning.
Revised: Various types of intermediate visual representations have been explored for visuomotor policy learning.

Original: Object bounding boxes have been commonly used as intermediate visual representations [23–25].
Revised: Object bounding boxes have been commonly used as intermediate visual representations [28, 16, 29].

Original: However, they require ﬁne-tuned orcategory-speciﬁc detectors and cannot easily generalize to tasks with previously unknown objects.
Revised: However, they require ﬁne-tuned or category-speciﬁc detectors and cannot easily generalize to tasks with previously unknown objects.

Original: Recently, deep learning methods enable IL to train policies end-to-end on raw observations [2, 3].
Revised: Recently, deep learning methods have enabled IL to train policies end-to-end on raw observations [2, 3].

Original: These methods are prone to covariate shift and causal confusion [5], resulting in poor generalizationperformances.
Revised: These methods are prone to covariate shift and causal confusion [6], resulting in poor generalization performances.

Original: Similar to our work, a large body of literature has looked into incorporating additionalinductive biases into end-to-end policies.
Revised: Similar to our work, a large body of literature has looked into incorporating additional inductive biases into end-to-end policies.

Original: Notable ones include spatial attention [26–28] and affordances [29, 30].
Revised: Notable ones include spatial attention [30–32] and affordances [33–38].

Original: However, these representations are purposefully designed for speciﬁc motion primitives, such as pick-and-place, limiting their abilities to generate diverse manipulation behaviors.
Revised: However, these representations are purposefully designed for speciﬁc motion primitives, such as pick-and-place, limiting their abilities to generate diverse manipulation behaviors.

Original: Object-Centric Representation.
Revised: Object-Centric Representation.

Original: Object-centric representations have been widely used in visualunderstanding and robotics tasks, where researchers seek to reason about visual scenes in a modularway based on the objects presented.
Revised: Object-centric representations have been widely used in visual understanding and robotics tasks, where researchers seek to reason about visual scenes in a modular way based on the objects presented.

Original: In robotics, poses [31–33] or bounding boxes [23–25] arecommonly used as object-level abstractions.
Revised: In robotics, poses [39–41] or bounding boxes [28, 16, 29] are commonly used as object-level abstractions.

Original: These representations often require prior knowledgeabout object instance/category and do not capture ﬁne-grained details, falling short in applying tonew tasks without previous unknown objects.
Revised: These representations often require prior knowledge about object instance/category and do not capture ﬁne-grained details, falling short in applying to new tasks without previous unknown objects.

Original: Unsupervised object discovery methods [34, 35] learnobject representation without manual supervision.
Revised: Unsupervised object discovery methods [42, 43] learn object representation without manual supervision.

Original: However, they fall short in handling complexscenes [34, 35], hindering the applicability to realistic manipulation tasks.
Revised: However, they fall short in handling complex scenes [42, 43], hindering the applicability to realistic manipulation tasks.

Original: Recent work from thevision community has made great progress in generating object proposals for various downstreamtasks, such as object detection [14–16] and visual-language reasoning [36, 37].
Revised: Recent work from the vision community has made signiﬁcant progress in generating object proposals for various downstream tasks, such as object detection [19–21] and visual-language reasoning [44, 45].

Original: Motivated by theeffectiveness of region proposal networks (RPNs) on out-of-distribution images [16], we use objectproposals to scaffold our object-centric representations for robot manipulation tasks.
Revised: Motivated by the effectiveness of region proposal networks (RPNs) on out-of-distribution images [21], we use object proposals to scaffold our object-centric representations for robot manipulation tasks.

Original:  Approach
Revised:  Approach

Original: We introduce VIOLA, an object-centric imitation learning approach to vision-based manipulation.
Revised: We introduce VIOLA, an object-centric imitation learning approach to vision-based manipulation.

Original: The core idea is to factorize raw visual observations into object-centric representations, on top ofwhich the policy generates closed-loop control actions.
Revised: The core idea is to decompose raw visual observations into object-centric representations, on top of which the policy generates closed-loop control actions.

Original: Figure 2 illustrates the pipeline.
Revised: Figure 2 illustrates the pipeline.

Original: We ﬁrstformulate the problem of visuomotor policy learning and describe two key components of VIOLA:1) how we build the object-centric representations based on general object proposals, and 2) how weuse a transformer-based architecture to learn policy over the object-centric representations.
Revised: We ﬁrst formulate the problem of visuomotor policy learning and describe two key components of VIOLA: 1) how we build the object-centric representations based on general object proposals, and 2) how we use a transformer-based architecture to learn policy over the object-centric representations.

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: We formulate a robot manipulation task as a discrete-time Markov Decision Process, which is deﬁned as a 5-tuple: M = ( S , A , P , R,  ) , where S is the state space,
Revised: We formulate a robot manipulation task as a discrete-time Markov Decision Process, which is deﬁned as a 5-tuple: M = ( S , A , P , R,  ) , where S is the state space,

Original: A is the action space, P ( ·| s, a )is the stochastic transition probability, R ( s, a, s 0 ) is the reward function, and  2 [0 , 1) is the discount factor.
Revised: A is the action space, P ( ·| s, a ) is the stochastic transition probability, R ( s, a, s 0 ) is the reward function, and  2 [0 , 1) is the discount factor.

Original: In our context, S is the space of the robot’s raw sensory data, including RGB imagesand proprioception, A is the space of the robot’s motor commands, and ⇡ : S !
Revised: In our context, S is the space of the robot’s raw sensory data, including RGB images and proprioception, A is the space of the robot’s motor commands, and ⇡ : S !

Original: A is a closed-loopsensorimotor policy that we deploy on the robot to perform a task.
Revised: A is a closedloop sensorimotor policy that we deploy on the robot to perform a task.

Original: Our goal  is to learnthe policy ⇡from a set of human demonstrations that maximizes the expected return E [
Revised: The goal of learning a visuomotor policy for robot manipulation is to learn a policy ⇡ that maximizes the expected return E [

Original: P [-1 t =
-]
Revised: P [-1 t =
-][Equation]

Original:  t R ( s t , a t , s t +1 )] .
Revised: In our work, we use behavioral cloning as our imitation learning algorithm. We assume access to a set of N demonstrations D = { ⌧ i } Ni =1 , where each trajectory ⌧ i is demonstrated through teleopera- tion. The goal of our behavioral cloning approach is to learn a policy that clones the actions from demonstrations D .

Original: We aim to design an object-centric representation that factorizes the visual observations of an unstructured scene into features of individual entities.
Revised: We aim to design an object-centric representation that factorizes the visual observations of an unstructured scene into features of individual entities.

Original: For vision-based manipulation, we assume noaccess to the ground-truth states of object.
Revised: For vision-based manipulation, we assume no access to the ground-truth states of objects.

Original: Rather, we use the top K object proposals from a pretrained Region Proposal Network (RPN) [13] to represent the set of object-related regions of an image.
Revised: Instead, we use the top K object proposals from a pre-trained Region Proposal Network (RPN) [18] to represent the set of object-related regions of an image.

Original: These proposals are grounded on image regions and optimized for covering the bounding boxesof potential objects.
Revised: These proposals are grounded on image regions and optimized for covering the bounding boxes of potential objects.

Original: We treat these K proposals as the K (approximate) objects and extract visualand positional features to build region features from each proposal.
Revised: We treat these K proposals as the K (approximate) objects and extract visual and positional features to build region features from each proposal.

Original: For manipulation tasks, reasoning about object interactions is also important to decide actions, To provide contextual informationfor such relational reasoning, we design three context features : a global context feature from theworkspace image to encode the current stage of the task; an eye-in-hand visual feature from the eyein-hand camera to mitigate occlusion and partial observability of objects; and a proprioceptive feature from the robot’s states.
Revised: For manipulation tasks, reasoning about object interactions is also essential to decide actions, To provide contextual information for such relational reasoning, we design three context features : a global context feature from the workspace image to encode the current stage of the task; an eye-in-hand visual feature from the eyein-hand camera to mitigate occlusion and partial observability of objects; and a proprioceptive feature from the robot’s states.

Original: We call the set of region features and context features at time step t as the
Revised: We call the set of region features and context features at time step t as the

Original: Action Token
Revised: Action Token

Original: Transformer
Revised: Transformer

Original: GMMMLP
Revised: GMMMLP

Original: VIOLA
Revised: VIOLA

Original: TemporalPosition 20Hz s t s t  H
Revised: Temporal Position 20Hz s t s t  H

Original: Object-CentricRepresentation
Revised: Object-Centric Representation

Original: FeatureEncoder 

Revised: FeatureEncoder

Original: H z t a t
Revised: H z t a t

Original: Feature Encoder
Revised: Feature Encoder

Original: Spatial Feature Map
Revised: Spatial Feature Map

Original: Proprioception
Revised: Proprioception

Original: Eye-in-hand Image
Revised: Eye-in-hand Image

Original: GlobalContext
Revised: GlobalContext

Original: FeaturePooling
Revised: FeaturePooling

Original: BoundingBoxPositions
Revised: BoundingBoxPositions

Original: RPN
Revised: RPN

Original: ResNet-18 Workspace Image h t
Revised: ResNet-18 WorkspaceImage h t

Original: FeatureEncoder
Revised: FeatureEncoder

Original: Task-Space Command
Revised: Task-Space Command

Original: SinusoidalEncoding
Revised: Sinusoidal Encoding

Original: Addition
Revised: Addition

Original: Contextfeatures
Revised: Context features

Original: Regionfeatures
Revised: Region features

Original: Per-StepFeatures
Revised: Per-StepFeatures

Original: H+1StepsofObservations
Revised: H+1 Steps of Observations

Original: RegionFeatures
Revised: RegionFeatures

Original: Concatenation
Revised: Concatenation

Original: TopKProposals
Revised: TopKProposals

Revised: SpatialSoftmax

Original: [Figure] ResNet [Figure] h 
Revised: ResNetLinearProjection h t h t  H

Original: [Figure]
Revised: [Figure]

Original: h
Revised: a

Original: [Figure]

Original: ˆ a

Original: [Figure]
Revised: [Figure]

Original: per-step feature h t .
Revised: per-step feature h t .

Original: Finally, our object-centric representation z t is a composition of per-step featuresfrom the last H + 1 time steps with their temporal positional encoding, as we describe in Sec. 3.2.
Revised: Finally, our object-centric representation z t is a composition of per-step features from the last H + 1 time steps with their temporal positional encoding, as we describe in Sec. 3.2.

Original: 3.2 Object-Centric Representation
Revised: 3.2 Object-Centric Representation

Original: This section describes how to build the object-centric representation z t .
Revised: This section describes how to build the object-centric representation z t .

Original: VIOLA ﬁrst obtains generalobject proposals using a pre-trained RPN.
Revised: VIOLA ﬁrst obtains general object proposals using a pre-trained RPN.

Original: VIOLA computes region features from proposals and obtains a per-step feature h t using the region features and the three context features which we describenext.
Revised: VIOLA computes region features from proposals and obtains a per-step feature h t using the region features and the three context features which we describe next.

Original: VIOLA builds z t through the temporal composition of h t from a history of observations.
Revised: VIOLA builds z t through the temporal composition of h t from a history of observations.

Original: General Object Proposals.
Revised: General Object Proposals.

Original: At each time step t , we generate object proposals using a pre-trained
Revised: At each time step t , we generate object proposals using a pre-trained

Original: RPN on the workspace image.
Revised: RPN on the workspace image.

Original: We select the top K proposals with the highest conﬁdence scores,which indicate regions that contain objects with the highest likelihood.
Revised: We select the top K proposals with the highest conﬁdence scores, which indicate regions that contain objects with the highest likelihood.

Original: The intuition of using a pretrained RPN is that it captures good priors of “objectness” in RGB images through  supervision ofnatural image datasets.
Revised: The intuition of using a pretrained RPN is that it captures good priors of “objectness” in RGB images through the supervision of natural image datasets.

Original: Recent work shows that an RPN trained on an 80-class dataset can generalizeto a 2000-class dataset [16].
Revised: Recent work shows that an RPN trained on an 80-class dataset can generalize to a 2000-class dataset [21].

Original: Our preliminary studies suggested that such a generalization ability alsoholds for localizing objects on raw images in our simulation and real-world tasks regardless of thedomain gaps, and we use the pre-trained RPN from Zhou et al.
Revised: Our preliminary studies suggested that such a generalization ability also holds for localizing objects on raw images in our simulation and real-world tasks regardless of the domain gaps. We use the pre-trained RPN from Zhou et al.

Original: [16] to localize regions with objects.
Revised: [21] to localize regions with objects.

Original: Region Features.
Revised: Region Features.

Original: For our policy to reason over objects and their spatial relations, we need to identify what objects each region contains and where these regions are from the top K proposals.
Revised: For our policy to reason over objects and their spatial relations, we need to identify what objects each region contains and where these regions are from the top K proposals.

Original: Toencode this information, we design a visual feature and a positional feature for each region.
Revised: To encode this information, we design a visual feature and a positional feature for each region.

Original: Toextract the visual feature from a region, we learn a spatial feature map by encoding the workspaceimage with the ResNet-18 module [38] and extract the visual feature using ROI Align [12].
Revised: To extract the visual feature from a region, we learn a spatial feature map by encoding the workspace image with the ResNet-18 module [46] and extract the visual feature using ROI Align [17].

Original: Weuse a learned spatial feature map for visual features, rather than from a pre-trained  detection modelbecause we share the same objective of localizing objects as object detectors, but different goals forthe downstream tasks — pre-trained visual models are optimized for visual recognition tasks but weneed actionable visual features that are informative for continuous control.
Revised: We use a learned spatial feature map for visual features rather than from a pre-trained feature pyramid network in detection models because we share the same objective of localizing objects as object detectors but different goals for the downstream tasks — pre-trained feature pyramid networks are optimized for visual recognition tasks, but we need actionable visual features that are informative for continuous control.

Revised: Such a design choice is supported by our ablation experiment in Appendix B.

Original: For positional features,we encode the coordinates of bounding box corners on images using sinusoidal positional encoding [17] (more details in Appendix A).
Revised: For positional features, we encode the coordinates of bounding box corners on images using sinu- soidal positional encoding [22] (more details in Appendix A).

Original: We ﬂatten each visual feature and add it to the positionalfeature of the same region to obtain a region feature.
Revised: We ﬂatten each visual feature and add it to the positional feature of the same region to obtain a region feature.

Original: Context Features.
Revised: Context Features.

Original: We extract the region features for the policies to reason over individual objects.
Revised: We extract the region features for the policies to reason over individual objects.

Original: However, they are insufﬁcient for decision-making in vision-based manipulation tasks, so we intro- duce three context features to assist decision-making.
Revised: However, they are insufﬁcient for decision-making in vision-based manipulation tasks, so we introduce three context features to assist decision-making.

Original: As regions only encode local information, weadd a global context feature to encode the current task stage from observation, which is computedfrom the spatial feature map of the workspace image using Spatial Softmax.
Revised: As regions only encode local information, we add a global context feature to encode the current task stage from observation, which is computed from the spatial feature map of the workspace image using Spatial Softmax.

Original: During manipulation,the robot’s gripper often occludes objects in the workspace view, so we add an eye-in-hand featurefrom eye-in-hand images to represent the occluded objects.
Revised: During manipulation, the robot’s gripper often occludes objects in the workspace view, so we add an eye-in-hand feature from eye-in-hand images to represent the occluded objects.

Original: We also encode the robot’s measurements of its joints and the gripper into proprioceptive features for the policy to generate preciseactions based on the robot’s states.
Revised: We also encode the robot’s measurements of its joints and the gripper into proprioceptive features for the policy to generate precise actions based on the robot’s states.

Original: We aggregate these contextfeatures and region features at eachtime step t into a set, which we refer to as per-step feature h t .
Revised: We aggregate the context and region features at each time step t into a set, which we refer to as per-step feature h t .

Original: Temporal Composition.
Revised: Temporal Composition.

Original: We build object-centric representation z t through  temporal compositionof h t that captures temporal dependencies and dynamic changes of object states.
Revised: We build object-centric representation z t through the temporal composition of h t that captures temporal dependencies and dynamic changes of object states.

Original: Building policiesover a sequence of past observations rather than the most recent observation has been shown effectiveby prior work [1, 22].
Revised: Building policies over a sequence of past observations rather than the most recent observation has been shown effective by prior work [1, 27].

Original: In our work,  temporal composition also increases the recall rate of objectproposals, making the policy more robust in the case of detection failures.
Revised: In our work, the temporal composition also increases the recall rate of object proposals, making the policy more robust in the case of detection failures.

Original: More speciﬁcally, z tis built from a set of per-step features { h t  i } Hi =0 from the last H + 1 time steps.
Revised: More speciﬁcally, z t is built from a set of per-step features { h t  i } Hi =0 from the last H + 1 time steps.

Original: To encode theirtemporal ordering, we add sinusoidal position encoding of the temporal positions { PE i } Hi =0 to theper-step features, resulting in our object-centric representations z t = { h t  i  PE i } Hi =0 .
Revised: To encode their temporal ordering, we add sinusoidal position encoding of the temporal positions { PE i } Hi = to the per-step features, resulting in our object-centric representation z t = { h t  i  PE i } Hi =0 .

Original: Our ablativestudies (see Sec. 4) indicate the importance of temporal positional encoding that retains with thetemporal ordering of features, especially when using our transformer-based policy.
Revised: Our ablative studies (see Sec. 4) indicate the importance of temporal positional encoding that retains the temporal ordering of features, especially when using our transformer-based policy.

Original: 3.3 Transformer-based Policy
Revised: 3.3 Transformer-based Policy

Original: We desire a policy that focuses on task-relevant region features in z t to generate actions.
Revised: We desire a policy that focuses on task-relevant region features in z t to generate actions.

Original: Regions that associate with task-relevant objects facilitate the accurate prediction of actions whileregions with task-irrelevant objects are likely to confound the policies.
Revised: Regions that associate with task-relevant objects facilitate the accurate prediction of actions, while regions with task-irrelevant objects are likely to confound the policies.

Original: We seek to use a transformer [17] as the policy backbone, which allows policies to reason over objects and their relations with its self-attention mechanism.
Revised: We seek to use a transformer [22] as the policy backbone, which allows policies to reason over objects and their relations with its selfattention mechanism.

Original: The core of a transformer is an encoder layer which consists of a multi-head self-attention block (MHSA), a layer-normalization [39] function, and a fullyconnected neural network (FFN).
Revised: The core of a transformer is an encoder layer which consists of a multi-head self-attention block (MHSA), a layer-normalization [47] function, and a fully-connected neural network (FFN).

Original: A transformer encoder layer takes as input a sequence of n latentvectors ( x 1 , . . .
Revised: A transformer encoder layer takes as input a sequence of n latent vectors ( x 1 , . . .

Original: , x n ) (also called tokens ) and outputs a sequence of n latent vectors ( y 1 , . .
Revised: , x n ) (also called tokens ) and outputs a sequence of n latent vectors ( y 1 , . .

Original: y n ) .
Revised: y n ) .

Original: A multi-head self-attention (MHSA) block consists of multiple self-attention blocks in parallel,which computes attention over all the tokens and compute a weighted sum of token values.
Revised: A MHSA block consists of multiple self-attention blocks in parallel, which computes attention over all the tokens and computes a weighted sum of token values.

Original: Werefer readers to Vaswani et al.
Revised: We refer readers to Vaswani et al.

Original: for more details [17].
Revised: for more details [22].

Original: In short, each transformer encoder outputs Y = FFN ( LayerNorm ( MSHA ( X )) , where each row ofY is an output latent vectors
Revised: In short, each transformer encoder outputs Y = FFN ( LayerNorm ( MSHA ( X )) , where each row of

Original:  y i that corresponds to x i .
Revised: Y is an output latent vectors y i that corresponds to x i .

Original: Our transformer-based policy is a stack of multiple transformer encoder layers, whichallows for a higher degree of expressiveness over input tokens compared to a single layer.
Revised: Our transformer-based policy is a stack of multiple transformer encoder layers, which allows for a higher degree of expressiveness over input tokens compared to a single layer.

Original: For our policy, we tokenize our object-centric representation z t , treating each region and contextfeature vector as an input token.
Revised: For our policy, we tokenize our object-centric representation z t , treating each region and context feature vector as an input token.

Original: To make action generation attend more to task-relevant regionfeatures than task-irrelevant ones, we append a learnable token, action token, to the input sequenceof transformer.
Revised: To make action generation attend more to task-relevant region features than task-irrelevant ones, we append a learnable token, action token, to the input sequence of a transformer.

Original: The action token design resembles the speciﬁc class tokens in natural languageunderstanding tasks [40] or visual recognition tasks [41], which are used for outputting latentvectors for downstream tasks.
Revised: The action token design resembles the speciﬁc class tokens in natural language understanding tasks [48] or visual recognition tasks [49], which are used for outputting latent vectors for downstream tasks.

Original: Similarly, we can get the output latent vector ˆ a t of the actiontoken, which learn to attend to task-relevant regions through action supervision.
Revised: Similarly, we can get the output latent vector ˆ a t from the action token, which learns to attend to task-relevant regions through action supervision.

Original: In the end, wepass ˆ a t through a two-layered fully-connected network, followed by a GMM (Gaussian Mixture
Revised: In the end, we pass ˆ a t through a two-layered fully-connected network, followed by a GMM (Gaussian Mixture

Original: Model) output head, which has been shown effective to capture the diverse multimodal behaviors indemonstration data [42, 22].
Revised: Model) output head, which has been shown effective to capture the diverse multimodal behaviors in demonstration data [50, 27].

Original: Implementation Details.
Revised: Implementation Details.

Original: Here we describe several key implementation details.
Revised: Here we describe several key implementation details.

Original: Full implementation details are in Appendix A.
Revised: Full implementation details are in Appendix A.

Original: We set K = 20 in simulation which offers a nice trade-off betweenobject recall rates and computation efﬁciency (see Appendix B.1).
Revised: We set K = 20 in simulation, which offers a nice trade-off between object recall rates and computation efﬁciency (see Appendix B.1).

Original: In the real world, we set K = 15which are found to attain a high recall rate on real images.
Revised: In the real world, we set K =to attain a high recall rate on real images.

Original: For temporal composition, we use H = 9to be consistent with the setup in BC-RNN [22].
Revised: For temporal composition, we use H = 9 to be consistent with the setup in BC-RNN [27].

Original: The action token is a learnable parameter randomlyinitialized from  normal distribution.
Revised: The action token is a learnable parameter randomly initialized from the normal distribution.

Original: We use data augmentation durining training to encourage thegeneralization ability of policies.
Revised: We use color augmentation and pixel shifting [4, 27, 49] during training to encourage the generalization ability of policies.

Original: We adopt random erasing [43] as well as standard pixel shifting [4, 22, 41] to prevent policies from overﬁtting to speciﬁc region features. 
Revised: Additionally, we adopt random erasing [51] to prevent policies from overﬁtting to speciﬁc region features. Concretely, we randomly apply random erasing during

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: training with a probability of 0 . 5 .

Revised: Random erasing ﬁlls in Gaussian noise into a randomly selected region whose size is 2 % to 5 % of the image with an aspect ratio ranging from 0 .

Revised: 5 to 1 . 5 .

Original:  Experiments
Revised:  Experiments

Original: We design our experiments to answer the following questions: 1) How well does VIOLA performagainst state-of-the-art end-to-end imitation learning algorithms?
Revised: We design our experiments to answer the following questions: 1) How well does VIOLA perform against state-of-the-art end-to-end imitation learning algorithms?

Original: How does it take advantage ofobject-centric representations?
Revised: How does it take advantage of object-centric representations?

Original: What design choices are essential for good performance?
Revised: What design choices are essential for good performance?

Original: and 4)
Revised: and 4)

Original: Is VIOLA practical for real-world deployment?
Revised: Is VIOLA practical for real-world deployment?

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: We conduct quantitative evaluations in simulation and real-world tasks to validate our approach.
Revised: We conduct quantitative evaluations in simulation and real-world tasks to validate our approach.

Original: Thesimulation tasks are designed with the robosuite [44] framework and used for quantitative comparisons between VIOLA and baselines.
Revised: The simulation tasks are designed with the robosuite [52] framework and used for quantitative comparisons between VIOLA and baselines.

Original: We also validate our design choices through ablative studies.
Revised: We also validate our design choices through ablative studies.

Original: We design three simulation tasks, Sorting , Stacking , BUDS-Kitchen , and three real-world tasks,
Revised: We design three simulation tasks, Sorting , Stacking , BUDS-Kitchen , and three real-world tasks,

Original: Dining-PlateFork , Dining-Bowl , Make-Coffee that cover a rich spectrum of manipulation behaviors that combine prehensile and non-prehensile motions.
Revised: Dining-PlateFork , Dining-Bowl , Make-Coffee that cover a rich spectrum of manipulation behaviors that combine prehensile and non-prehensile motions.

Original: We visualize their initial and goal conﬁgurations in Figure 3.
Revised: We visualize their initial and goal conﬁgurations in Figure 3.

Original: We design these tasks to understand if the use of object priors beneﬁts policylearning along two axes of task characteristics:
Revised: We design these tasks to understand if the use of object priors beneﬁts policy learning along two axes of task characteristics:

Original: large placement variations of objects and multi-stagelong-horizon execution.
Revised: large placement variations of objects and multi-stage long-horizon execution.

Original: For the ﬁrst axis, we design Sorting and Stacking to have large initialranges of object placements.
Revised: For the ﬁrst axis, we design Sorting and Stacking to have large initial ranges of object placements.

Original: For the second axis, we use BUDS-Kitchen , a multi-stage long-horizonmanipulation task from prior work [4].
Revised: For the second axis, we use BUDS-Kitchen , a multi-stage long-horizon manipulation task from prior work [4].

Original: The real-world tasks are designed to resemble real-world everyday tasks: Dining-PlateFork and Dining-Bowl for dining table arrangements, Make-Coffeefor espresso coffee making.
Revised: The real-world tasks are designed to resemble real-world everyday tasks: Dining-PlateFork and Dining-Bowl for dining table arrangement, Make-Coffee for espresso coffee making.

Original: Full details of tasks and data collection are included in Sec.
Revised: Full details of tasks and data collection are included in Sec.

Original: C.
Revised: C.

Original: Evaluation setup To systematically evaluate the efﬁcacy and robustness of the learned policies,we design the following testing setups in simulation: 1) Canonical : all the objects and sensorconﬁgurations follow the same distribution as seen in demonstrations;  2) Three testing variants,namely Placement , Distractor , and Camera-Jitter .
Revised: Evaluation Setup. To systematically evaluate the efﬁcacy and robustness of the learned policies, we design the following testing setups in simulation: 1) Canonical : all the objects and sensor conﬁgurations follow the same distribution as seen in demonstrations; and 2) Three testing variants, namely Placement , Distractor , and Camera-Jitter .

Original: The design of these variants is inspiredby the MAGICAL benchmark [45] for evaluating systematic generalization of imitation learningpolicies.
Revised: The design of these variants is inspired by the MAGICAL benchmark [53] for evaluating systematic generalization of imitation learning policies.

Original: The design principle of testing variants is to keep the task semantics the same as in thecanonical setup, and we identify the three challenging axes of variations for learning-based manipulation, namelyobject initial  placements ( Placement ), distractor objects presented in the scene( Distractor ), and camera pose jitters ( Camera-Jitter ).
Revised: The design principle of testing variants is to keep the task semantics the same as in the canonical setup. At the same time, we identify the three challenging axes of variations for learningbased manipulation, namely initial object placements ( Placement ), distractor objects presented in the scene ( Distractor ), and camera pose jitters ( Camera-Jitter ).

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4.2 Experiment Results
Revised: 4.2 Experiment Results.

Original: We answer question (1) by quantitatively comparing VIOLA with the following baselines: BC : behavioral cloning that conditions on current observations; OREO [5]: behavioral cloningmethod that learns object-aware discrete codes using VQ-VAE [47] and learns policies thereon; BC-RNN [22]: a state-of-the-art method that uses a temporal sequence of past observations withrecurrent neural networks.
Revised: We answer question (1) by quantitatively comparing VIOLA with the following baselines: BC : behavioral cloning that conditions on current observations; OREO [6]: behavioral cloning method that learns object-aware discrete codes using VQ-VAE [55] and learns policies thereon; BCRNN [27, 56, 15, 57]: a state-of-the-art method that uses a temporal sequence of past observations with recurrent neural networks.

Original: VIOLA-Patch : a variant of VIOLA, where we use a regular gridof image patches as inputs to the policy rather than proposals, similar to ViT [48].
Revised: VIOLA-Patch : a variant of VIOLA, where we use a regular grid of image patches as inputs to the policy rather than proposals, similar to ViT [58].

Original: To make fairevaluation, we randomly generate 100 initial conﬁgurations and repeat runs with three differentrandom seeds.
Revised: To make a fair evaluation, we randomly generate 100 initial conﬁgurations and repeat runs with three different random seeds.

Original: We evaluate all policies on the same set of pre-generated initial conﬁgurations.
Revised: We evaluate all policies on the same set of pre-generated initial conﬁgurations.

Original: The result in Table 1 shows that VIOLA outperforms the most competitive baseline BC-RNN by50 . 8% success rate in Canonical and 44 .
Revised: Table 1 shows that VIOLA outperforms the most competitive baseline BC-RNN by 50 . 8% success rate in Canonical and 44 .

Original: 1% in three testing variants.
Revised: 1% in three testing variants.

Original: VIOLA’s strong performancesuggests the advantage of object-centric representations for visuomotor imitation.
Revised: VIOLA’s strong performance suggests the advantage of object-centric representations for visuomotor imitation.

Original: OREO’s resultssuggest that learning object-aware discrete codes via unsupervised learning does not consistently improve performance over simple BC for all tasks.
Revised: OREO’s results suggest that learning object-aware discrete codes via unsupervised learning does not consistently improve performance over simple BC for all tasks.

Original: The comparisons between BC, OREO, and the othermethods using temporal windows show the importance of temporal modeling for these algorithmsto achievethe high performances and robustness in complex vision-based manipulation domains.
Revised: The comparisons between BC, OREO, and the other methods using temporal windows show the importance of temporal modeling for these algorithms to achieve high performances and robustness in complex vision-based manipulation domains.

Original: Table 1 shows that VIOLA-Patch has comparable performance to VIOLA in some evaluation setups.
Revised: Table 1 shows that VIOLA-Patch has comparable performance to VIOLA in some evaluation setups.

Original: It suggests that the transformer backbone exhibits capability to attend to patch regions that cover task-relevant visual cues, even though the patch division is agnostic to objects.
Revised: It shows that the transformer backbone can attend to patch regions that cover task-relevant visual cues, even though the patch division is agnostic to objects.

Original: Nonetheless, VIOLA still performs better than VIOLA-Patch, especially in the long-horizon task BUDS-Kitchen , suggesting regions with complete coverage of objects is critical for the success of VIOLA, highlighting the importance of the object priors.
Revised: Nonetheless, VIOLA still performs better than VIOLA-Patch, especially in the long-horizon task BUDS-Kitchen , suggesting regions with complete coverage of objects is critical for the success of VIOLA, highlighting the importance of the object priors.

Original: Ablative Studies To answer question (2) and (3), we use ablative studies to validate our model’sdesign and show how it takes advantage of the object-centric priors.
Revised: Ablative Studies. To answer questions (2) and (3), we use ablative studies to validate our model’s design and show how it takes advantage of the object-centric priors.

Original: The results in Table 2 quantifythe effects of the transformer backbone, the use of temporal windows,the use of object proposalregions, and our data augmentation technique.
Revised: Table 2 quantiﬁes the effects of the transformer backbone, the use of temporal windows, object proposal regions, and our data augmentation technique.

Original: We start with our base model, a transformermodel that only takes the current observations as input.
Revised: We start with our base model, a transformer model that only takes the current observations as input.

Original: Second, we add the temporal windowof observations similar to BC-RNN.
Revised: Second, we add the temporal window of observations similar to BC-RNN.

Original: Results show that the use of temporal observation is keyto unleashing the power of the transformer architecture, making the performance of this ablatedversion comparable to BC-RNN.
Revised: Results show that the use of temporal observation is key to unleashing the power of the transformer architecture, making the performance of this ablated version comparable to BC-RNN.

Original: Nonetheless, this model does not encode temporal ordering as thetransformer model is invariant to input permutation.
Revised: Nonetheless, this model does not encode temporal ordering as the transformer model is invariant to input permutation.

Original: In the next row, we add temporal positionalencoding [17] to the input sequence, which produces a method that outperforms the top baselines.
Revised: In the next row, we add temporal positional encoding [22] to the input sequence, which produces a method that outperforms the top baselines.

Original: In the next two rows, we procedurally addin visual and positional features of regions to prove thatour model does exploit the object priors.
Revised: In the next two rows, we procedurally add visual and positional features of regions to prove that our model does exploit the object priors.

Original: Results show that visual features alone only improve ourmodel’s robustness to Distractor .
Revised: Results show that visual features alone only improve our model’s robustness to Distractor .

Original: When using both visual and positional features, the modelperforms better in Canonical and Distractor , but worse in Placement and Camera-Jitter .
Revised: When using both visual and positional features, the model performs better in Canonical and Distractor , but worse in Placement and Camera-Jitter .

Original: We hypothesize that this ablated version overﬁts to locations of proposal boxes in demonstrations,therefore it generalizes worse in Placement and Camera-Jitter where the distribution of objectlocations on 2D images shift from demonstrations.
Revised: We hypothesize that this ablated version overﬁts to locations of proposal boxes in demonstrations; therefore, it generalizes worse in Placement and Camera-Jitter where the distribution of object locations on 2D images shifts from demonstrations.

Original: To mitigate such overﬁtting, we use Random
Revised: To mitigate such overﬁtting, we use Random

Original: Erasing [43] to achieve  highest success rates in all evaluation setup. [Figure]
Revised: Erasing [51] to achieve the highest success rates in all evaluation setups.

Original: Real Robot Evaluation We compare VIOLA against the SOTA baseline, BC-RNN, on all threereal-world tasks.
Revised: Real Robot Evaluation. We compare VIOLA against the SOTA baseline, BC-RNN, on all three real-world tasks.

Original: We evaluate in 10 different initial conﬁgurations and repeat 3 times in the sameconﬁguration for each policy evaluation.
Revised: We evaluate in 10 different initial conﬁgurations and repeat 3 times in the same conﬁguration for each policy evaluation.

Original: To mitigate potential human bias introduced by setting upobjects before each trial, we employ the A/B testing paradigm for evaluation, that is, we reset theenvironment while being agnostic to the next policy to execute, and the whole evaluation processfor one initialization is repeated until all evaluation trials are completed.
Revised: To mitigate potential human bias introduced by setting up objects before each trial, we employ the A/B testing paradigm for evaluation. That is, we reset the environment while being agnostic to the next policy to execute. The entire evaluation process for one initialization is repeated until all evaluation trials are completed.

Original:  The quantitative evaluation in Figure 4 shows that VIOLA outperforms BC-RNN by 46 .
Revised: [Figure] The quantitative evaluation in Figure 4 shows that VIOLA outperforms BC-RNN by 46 .

Original: 7% successrate on average.
Revised: 7% success rate on average.

Original: Qualitatively, we observe that VIOLA can robustly grasp k-cups or open thecoffee machine in the Make-Coffee task, while BC-RNN tends to reach to wrong positions wheregrasps are missed, or to release the gripper mistakenly.
Revised: Qualitatively, we observe that VIOLA can robustly grasp k-cups or open the coffee machine in the Make-Coffee task, while BC-RNN tends to reach to wrong positions where grasps are missed or to release the gripper mistakenly.

Original: In the supplementary videos, we includethe evaluation rollouts of VIOLA for all the tasks.
Revised: In the supplementary videos, we include the evaluation rollouts of VIOLA for all the tasks.

Original: We also qualitatively visualize the attention of
Revised: We also qualitatively visualize the attention of

Original: VIOLA in Figure 5 by showing the top 3 regions with  highest attention weights at each timestepof temporal observation.
Revised: VIOLA in Figure 5 by showing the top 3 regions with the highest attention weights at each timestep of temporal observation.

Original: The ﬁgure shows that when the robot is to grasp  k-cup, VIOLA’s topattention is over the k-cup while it also takes  robot ﬁngers and coffee machine into account tofacilitate spatial reasoning.
Revised: The ﬁgure shows that when the robot is to grasp the k-cup, VIOLA’s top attention is over the k-cup while it also takes the robot ﬁngers and coffee machine into account to facilitate spatial reasoning.

Original: When the robot is to close the coffee machine, VIOLA’s top-1 attentionspreads over the k-cup, robot gripper, and the coffee machine, and the policy generates actions forclosing through spatial reasoning over these regions.
Revised: When the robot is to close the coffee machine, VIOLA’s top-1 attention spreads over the k-cup, robot gripper, and the coffee machine, and the policy generates actions for closing through spatial reasoning over these regions.

Original:  Conclusion
Revised:  Conclusion

Original: We present VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation.
Revised: We present VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation.

Original: Our approach uses general object proposals to build objectcentric representation.
Revised: Our approach uses general object proposals to build the object-centric representation.

Original: This representation encodes the visual and positional features from theproposal regions and context features of global scene information and robot states.
Revised: This representation encodes the visual and positional features from the proposal regions and context features of global scene information and robot states.

Original: We design atransformer-based policy to identify task-speciﬁc relevant regions for action generation.
Revised: We design a transformer-based policy to identify task-speciﬁc relevant regions for action generation.

Original: The resultsshow the superior performances of VIOLA compared to state-of-the-art baselines in simulation andthe real world.
Revised: The results show the superior performances of VIOLA compared to state-of-the-art baselines in simulation and the real world.

Original: We also validate our model designs through ablative studies, showing how eachcomponent of the model  impactsthe policy performance.
Revised: We also validate our model designs through ablative studies, showing how each model component impacts policy performance.

Original: Limitations While VIOLA has achieved great success in learning robust visuomotor policies, itsuffers from common limitations of learning from ofﬂine demonstration datasets.
Revised: Limitations. While VIOLA has achieved great success in learning robust visuomotor policies, it suffers from common limitations of learning from ofﬂine demonstration datasets.

Original: One future direction is to use this object-centric representation in an online manner so that it can improve overroll-out experience over time.
Revised: One future direction is to use this object-centric representation in an online manner so that it can improve over roll-out experience over time.

Original: Also, we use a pre-trained RPN without adaptation, it may fail to generalize to manipulation scenes with aggressive distributional shifts from natural images on whichthe RPN is trained.
Revised: Also, we use a pre-trained RPN without adaptation, which may fail to generalize to manipulation scenes with aggressive distributional shifts from natural images on which the RPN is trained.

Original: A possible improvement is to ﬁne-tune the RPN while training the policy.
Revised: A possible remedy is to ﬁne-tune the RPN while training the policy.

Original: Another limitation of our approach is that we only consider monocular RGB images in the input data,which lacks the 3D information of objects.
Revised: Another limitation of our approach is that we only consider monocular RGB images in the input data, which lacks the 3D information of objects.

Original: This formulation prevents VIOLA from excluding thebackground visual elements in the input, which would limit the generalization ability of VIOLA invariants of large visual changes, such as table texture change.
Revised: This formulation prevents VIOLA from excluding the background visual elements in the input, which would limit the generalization ability of VIOLA in variants of large visual changes, such as table texture change.

Original: For future work, we would like toextend VIOLA to using depth images so that distracting background elements such as tables can bedientangled from the representation.
Revised: For future work, we would like to extend VIOLA to using depth images so that distracting background elements such as tables can be disentangled from the object representations. Acknowledgments

Revised: The authors would like to thank Yue Zhao for the insightful discussion on the project and the manuscript.

Revised: This work has taken place in the Robot Perception and Learning Group (RPL) and

Revised: Learning Agents Research Group (LARG) at UT Austin.

Revised: RPL research has been partially supported by the National Science Foundation (CNS-1955523, FRR-2145283), the Ofﬁce of Naval Research (N00014-22-1-2204), and the Amazon Research Awards.

Revised: LARG research is supported in part by

Revised: NSF (CPS-1739964, IIS-1724157, FAIN-2019844), ONR (N00014-18-2243), ARO (W911NF-192-0333), DARPA, GM, Bosch, and UT Austin’s Good Systems grand challenge.

Revised: Peter Stone serves as the Executive Director of Sony AI America and receives ﬁnancial compensation for this work.

Revised: The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.