Original: MIRA: Mental Imagery for Robotic Affordances
Revised: MIRA: Mental Imagery for Robotic Affordances

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract: Humans form mental images of 3D scenes to support counterfactualimagination, planning, and motor control.
Revised: Abstract: Humans form mental images of 3D scenes to support counterfactual imagination, planning, and motor control.

Original: Our abilities to predict the appearance andaffordance of the scene from previously unobserved viewpoints aid us in performingmanipulation tasks (e.g., 6-DoF kitting) with a level of ease that is currently out of reachfor existing robot learning frameworks.
Revised: Our abilities to predict the appearance and affordance of the scene from previously unobserved viewpoints aid us in performing manipulation tasks (e.g., 6-DoF kitting) with a level of ease that is currently out of reach for existing robot learning frameworks.

Original: In this work, we aim to build artificial systemsthat can analogously plan actions on top of imagined images.
Revised: In this work, we aim to build artificial systems that can analogously plan actions on top of imagined images.

Original: To this end, we introduce
Revised: To this end, we introduce

Original: Mental Imagery for Robotic Affordances (MIRA), an action reasoning frameworkthat optimizes actions with novel-view synthesis and affordance prediction in the loop.
Revised: Mental Imagery for Robotic Affordances (MIRA), an action reasoning framework that optimizes actions with novel-view synthesis and affordance prediction in the loop.

Original: Given a set of 2D RGB images, MIRA builds a consistent 3D scene representation,through which we synthesize novel orthographic views amenable to pixel-wise affordances prediction for action optimization.
Revised: Given a set of 2D RGB images, MIRA builds a consistent 3D scene representation, through which we synthesize novel orthographic views amenable to pixel-wise affordances prediction for action optimization.

Original: We illustrate how this optimization processenables us to generalize to unseen out-of-plane rotations for 6-DoF robotic manipulationtasks given a limited number of demonstrations, paving the way toward machines thatautonomously learn to understand the world around them for planning actions.
Revised: We illustrate how this optimization process enables us to generalize to unseen out-of-plane rotations for 6-DoF robotic manipulation tasks given a limited number of demonstrations, paving the way toward machines that autonomously learn to understand the world around them for planning actions.

Original:  Introduction
Revised:  Introduction

Original: Suppose you are shown a small, unfamiliar object and asked if it could fit through an M-shaped slot.
Revised: Suppose you are shown a small, unfamiliar object and asked if it could fit through an M-shaped slot.

Original: Howmight you solve this task?
Revised: How might you solve this task?

Original: One approach would be to “rotate” the object in your mind’s eye and see if,from some particular angle, the object’s profile fits into an M. To put the object through the slot would thenjust require orienting it to that particular imagined angle.
Revised: One approach would be to “rotate” the object in your mind’s eye and see if, from some particular angle, the object’s profile fits into an M. To put the object through the slot would then just require orienting it to that particular imagined angle.

Original: In their famous experiments on “mental rotation”,
Revised: In their famous experiments on “mental rotation”,

Original: Shepard & Metzler argued that this is the approach humans use when reasoning about the relative posesof novel shapes [1].
Revised: Shepard & Metzler argued that this is the approach humans use when reasoning about the relative poses of novel shapes [1].

Original: Decades of work in psychology have documented numerous other ways that “mentalimages”, i e. pictures in our heads, can aid human cognition [2].
Revised: Decades of work in psychology have documented numerous other ways that “mental images”, i e. pictures in our heads, can aid human cognition [2].

Original: In this paper, we ask: can we give robotsa similar ability, where they use mental imagery to aid their spatial reasoning?
Revised: In this paper, we ask: can we give robots a similar ability, where they use mental imagery to aid their spatial reasoning?

Original: Fortunately, the generic ability to perform imagined translations and rotations of a scene, also knownas novel view synthesis , has seen a recent explosion of research in the computer vision and graphicscommunity [3, 4, 5].
Revised: Fortunately, the generic ability to perform imagined translations and rotations of a scene, also known as novel view synthesis , has seen a recent explosion of research in the computer vision and graphics community [3, 4, 5].

Original: Our work builds in particular upon Neural Radiance Fields (NeRFs) [6], which canrender what a scene would look like from any camera pose.
Revised: Our work builds in particular upon Neural Radiance Fields (NeRFs) [6], which can render what a scene would look like from any camera pose.

Original: We treat a NeRF as a robot’s “mind’s eye”,a virtual camera it may use to imagine how the scene would look were the robot to reposition itself.
Revised: We treat a NeRF as a robot’s “mind’s eye”, a virtual camera it may use to imagine how the scene would look were the robot to reposition itself.

Original: Wecouple this ability with an affordance model [7], which predicts, from any given view of the scene, whatactions are currently afforded.
Revised: We couple this ability with an affordance model [7], which predicts, from any given view of the scene, what actions are currently afforded.

Original: Then the robot must just search, in its imagination, for the mental imagethat best affords the action it wishes to execute, then execute the action corresponding to that mental image.
Revised: Then the robot must just search, in its imagination, for the mental image that best affords the action it wishes to execute, then execute the action corresponding to that mental image.

Original: We test this framework on 6-DoF rearrangement tasks [8], where the affordance model simply predicts,for each pixel in a given camera view, what is the action value of picking (or placing) at that pixel’scoordinates.
Revised: We test this framework on 6-DoF rearrangement tasks [8], where the affordance model simply predicts, for each pixel in a given camera view, what is the action-value of picking (or placing) at that pixel’s coordinates.

Original: Using NeRF as a virtual camera for this task has several advantages over prior works whichused physical cameras:
Revised: Using NeRF as a virtual camera for this task has several advantages over prior works which used physical cameras:

Original: • Out-of-plane rotation.
Revised: • Out-of-plane rotation.

Original: Prior works have applied affordance maps to 2-dimensional top-down cameraviews, allowing only the selection of top-down picking and placing actions [7, 9, 10].
Revised: Prior works have applied affordance maps to 2-dimensional top-down camera views, allowing only the selection of top-down picking and placing actions [7, 9, 10].

Original: We instead[Figure] formulate the pick and place problem as an action optimization process that searches across different novel synthesized views and their affordances of the scene.
Revised: We instead formulate the pick-and-place problem as an action optimization process that searches across different novel synthesized views and their affordances of the scene.

Original: We demonstrate that this optimizationprocess can handle the multi-modality of picking and placing while naturally supporting actions thatinvolve out-of-plane rotations. 
Revised: We demonstrate that this optimization process can handle the multi-modality of picking and placing while naturally supporting actions that involve out-of-plane rotations. [Figure] [Figure]

Original: • Orthographic ray casting.
Revised: • Orthographic ray casting.

Original: A NeRF trained with images from consumer cameras can be used tosynthesize novel views from novel kinds of cameras that are more suitable to action reasoning.
Revised: A NeRF trained with images from consumer cameras can be used to synthesize novel views from novel kinds of cameras that are more suitable to action reasoning.

Original: Mostphysical cameras use perspective projection, in which the apparent size of an object in the image planeis inversely proportional to that object’s distance from the camera — a relationship that any visionalgorithm must comprehend and disentangle.
Revised: Most physical cameras use perspective projection, in which the apparent size of an object in the image plane is inversely proportional to that object’s distance from the camera — a relationship that any vision algorithm must comprehend and disentangle.

Original: NeRF can instead create images under other renderingprocedures; we show that orthographic ray casting is particularly useful, which corresponds to anon-physical “camera” that is infinitely large and infinitely distant from the scene.
Revised: NeRF can instead create images under other rendering procedures; we show that orthographic ray casting is particularly useful, which corresponds to a non-physical “camera” that is infinitely large and infinitely distant from the scene.

Original: This yields images inwhich an object’s size in the image plane is invariant to its distance from the camera, and its appearanceis equivariant with respect to translation parallel to the image plane.
Revised: This yields images in which an object’s size in the image plane is invariant to its distance from the camera and its appearance is equivariant with respect to translation parallel to the image plane.

Original: In essence, this novel usage of NeRFallows us to generate “blueprints” for the scene that complement the inductive biases of algorithms thatencode translational equivariance (such as ConvNets).
Revised: In essence, this novel usage of NeRF allows us to generate “blueprints” for the scene that complement the inductive biases of algorithms that encode translational equivariance (such as ConvNets).

Original: • RGB-only.
Revised: • RGB-only.

Original: Prior rearrangement methods [11, 12, 13] commonly require 3D sensors (e.g. via structuredlight, stereo, or time-of-flight), and these are error-prone when objects contain thin structures or are composed of specular or semi-transparent materials—a common occurrence.
Revised: Prior rearrangement methods [11, 12, 13] commonly require 3D sensors (e g. via structured light, stereo, or time-of-flight), and these are error-prone when objects contain thin structures or are composed of specular or semi-transparent materials—a common occurrence (see Fig. 2 for examples).

Original: These limitations drastically restrict the set of tasks, objects, and surfaces these prior works can reason over.
Revised: These limitations drastically restrict the set of tasks, objects, and surfaces these prior works can reason over.

Original: We term our method Mental Imagery for Robotic Affordances, or MIRA.
Revised: We term our method Mental Imagery for Robotic Affordances, or MIRA.

Original: To test

Original:  MIRA, we perform experiments in both simulation and 
Revised: To test MIRA, we perform experiments in both simulation and the real world.

Original: the

Original: real

Original: world

Original: For simulation

Original: we

Original:  extend the Ravens  benchmark to include tasks that require 6-DoF actions
Revised: For simulation, we extend the Ravens [9] benchmark to include tasks that require 6-DoF actions.

Original: Our

Original:  model demonstrates superior performance to existing state- of-the-art methods for object rearrangement [14, 9] despite not requiring 
Revised: Our model demonstrates superior performance to existing stateof-the-art methods for object rearrangement [14, 9], despite not requiring depth sensors.

Original: depth

Original: sensors

Original: Importantly

Original: the

Original:  optimization process with novel view synthesis and affordance prediction in the loop enables our framework to generalize to out-of-distribution object configurations, where the baselines struggle
Revised: Importantly, the optimization process with novel view synthesis and affordance prediction in the loop enables our framework to generalize to out-of-distribution object configurations, where the baselines struggle.

Original: In summary, we contribute (i) a framework that uses NeRFs as the scene representation to perform novel view synthesis for precise object rearrangement, (ii) an orthographic ray casting procedure for NeRFs rendering that facilitates the policy’s translation 
Revised: In summary, we contribute (i) a framework that uses NeRFs as the scene representation to perform novel view synthesis for precise object rearrangement, (ii) an orthographic ray casting procedure for NeRFs rendering that facilitates the policy’s translation equivariance, (iii) an extended benchmark of 6-DoF manipulation tasks in Ravens [9], and (iv) empirical results on a broad range of

Original: equivariance, ( iii an extended benchmark of 6-DoF manipulation
Revised: manipulation

Original: tasks in Ravens [9],and (iv) empirical results on a broad range of manipulation tasks, validated with real-robot experiments.
Revised: tasks, validated with real- robot experiments.

Original:  Related Works 
Revised:  Related Works 2.1 Vision-based Manipulation.

Original: 2.1 Vision-based Manipulation.

Original: Object-centric.
Revised: Object-centric.

Original: Classical methods in visual perception for robotic manipulation mainly focus on representing instances with 6-DoF poses [15, 16, 17, 18, 19, 20, 21].
Revised: Classical methods in visual perception for robotic manipulation mainly focus on representing instances with 6-DoF poses [15, 16, 17, 18, 19, 20, 21].

Original: However, 6-DoF poses cannot representthe states of deformable objects or granular media, and cannot capture large intra-category variations ofunseen instances [11].
Revised: However, 6-DoF poses cannot represent the states of deformable objects or granular media, and cannot capture large intra-category variations of unseen instances [11].

Original: Alternative methods that represent objects with dense descriptors [22, 23, 24] orkeypoints [11, 25, 26, 27] improve generalization, but they require a dedicated data collection procedure(e.g. , configuring scenes with single objects).
Revised: Alternative methods that represent objects with dense descriptors [22, 23, 24] or keypoints [11, 25, 26, 27] improve generalization, but they require a dedicated data collection procedure (e.g., configuring scenes with single objects).

Original: Action-centric.
Revised: Action-centric.

Original: Recent methods based on end-to-end learning directly predict actions given visual observations [28, 29, 30, 31, 32, 33].
Revised: Recent methods based on end-to-end learning directly predict actions given visual observations [28, 29, 30, 31, 32, 33].

Original: These methods can potentially work with deformable objects or granularmedia, and do not require any object-specific data collection procedures.
Revised: These methods can potentially work with deformable objects or granular media, and do not require any object-specific data collection procedures.

Original: However, these methods areknown to be sample inefficient and challenging to debug.
Revised: However, these methods are known to be sample inefficient and challenging to debug.

Original: Recently, several works [9, 13, 34, 35, 36, 37, 38]have proposed to incorporate spatial structure into action reasoning for improved performance and bettersample efficiency.
Revised: Recently, several works [9, 13, 34, 35, 36, 37, 38] have proposed to incorporate spatial structure into action reasoning for improved performance and better sample efficiency.

Original: Among them, the closest work to ours is Song et al.
Revised: Among them, the closest work to ours is Song et al.

Original: [13] which relies on view synthesisto plan 6-DoF picking.
Revised: [13] which relies on view synthesis to plan 6-DoF picking.

Original: Our work differs in that it 1) uses NeRF whereas [13] uses TSDF [39], 2) doesnot require depth sensors, 3) uses orthographic image representation, 4) does not directly use the camerapose as actions, and 5) shows results on rearrangement tasks that require both picking and placing.
Revised: Our work differs in that it 1) uses NeRF whereas [13] uses TSDF [39], 2) does not require depth sensors, 3) uses orthographic image representation, 4) does not directly use the camera pose as actions, and 5) shows results on rearrangement tasks that require both picking and placing.

Original: 2.2 Neural Fields for Robotics
Revised: 2.2 Neural Fields for Robotics

Original: Neural fields have emerged as a promising tool to represent 2D images [40], 3D geometry [41, 42],appearance [6, 43, 44], touch [45], and audio [46, 47].
Revised: Neural fields have emerged as a promising tool to represent 2D images [40], 3D geometry [41, 42], appearance [6, 43, 44], touch [45], and audio [46, 47].

Original: They offer several advantages over classicrepresentations (e.g., voxels, point clouds, and meshes) including reconstruction quality, and memoryefficiency.
Revised: They offer several advantages over classic representations (e.g., voxels, point clouds, and meshes) including reconstruction quality, and memory efficiency.

Original: Several works have explored the usage of neural fields for robotic applications includinglocalization [48, 49], SLAM [50, 51, 52], navigation [53], dynamics modeling [54, 55, 56, 57], andreinforcement learning [58].
Revised: Several works have explored the usage of neural fields for robotic applications including localization [48, 49], SLAM [50, 51, 52], navigation [53], dynamics modeling [54, 55, 56, 57], and reinforcement learning [58].

Original: For robotic manipulation, GIGA [59] jointly trains a grasping network andan occupancy network for synergy.
Revised: For robotic manipulation, GIGA [59] and NDF [12] use occupancy networks’ feature fields to help action prediction.

Revised: However, both of them rely on depth cameras to perceive the 3D geometry of the scene, while MIRA does not require depth sensors and thus can handle objects with reflective or thin-structured materials.

Original: Dex-NeRF [60] infers the geometry of transparent objects with NeRFand determines the grasp poses with Dex-Net [30].
Revised: Dex-NeRF [60] infers the geometry of transparent objects with NeRF and determines the grasp poses with Dex-Net [30].

Original: NDF [12] uses the features of occupancy networks [42]as object descriptors for few-shot imitation learning. 
Revised: However, it only predicts the 3-DoF grasping pose and does not provide a solution for pick-conditioned placing; NeRF-Supervision [61] uses NeRF as a dataset generator to learn dense object descriptors for picking but not placing. In contrast, MIRA is capable of predicting both 6-DoF picking and pick-conditioned placing. Model-based methods [54, 55] use NeRFs as decoders to learn latent state representations for model predictive control; NeRF-RL [58] instead uses the learned latent state representations for downstream reinforcement learning. Compared to these works, MIRA uses imitation learning to acquire the policy and thus avoid the conundrum of constructing reward functions.

Original: NeRF-Supervision [61] uses NeRF as a datasetgenerator to learn dense object descriptors for picking.
Revised: Furthermore, MIRA enjoys better sample efficiency due to the approximate 3D rotational equivariance via novel-view synthesis. The last but not least, MIRA demonstrates real-world results on three 6-DoF kitting tasks while these works primarily focus on simulation benchmark.

Original:  Method
Revised:  Method

Original: Our goal is to predict actions a t , given RGB-only visual observations o t , and trained from only a limitednumber of demonstrations.
Revised: Our goal is to predict actions a t , given RGB-only visual observations o t , and trained from only a limited number of demonstrations.

Original: We parameterize our action space with two-pose primitives a t =( T pick , T place ) ,which are able to flexibly parameterize rearrangement tasks [9].
Revised: We parameterize our action space with two-pose primitives a t =( T pick , T place ) , which are able to flexibly parameterize rearrangement tasks [9].

Original: This problem is challenging due to thehigh degrees of freedom of a t (12 degrees of freedom for two full SE(3) poses), a lack of informationabout the underlying object state (such as object poses), and limited data.
Revised: This problem is challenging due to the high degrees of freedom of a t (12 degrees of freedom for two full SE(3) poses), a lack of information about the underlying object state (such as object poses), and limited data.

Original: Our method (illustrated in Fig. 1)factorizes action reasoning into two modules: 1) a continuous neural radiance field that can synthesizevirtual views of the scene at novel viewpoints, and 2) an optimization procedure which optimizes actionsby predicting per-pixel affordances across different synthesized virtual pixels.
Revised: Our method (illustrated in Fig. 1) factorizes action reasoning into two modules: 1) a continuous neural radiance field that can synthesize virtual views of the scene at novel viewpoints, and 2) an optimization procedure which optimizes actions by predicting per-pixel affordances across different synthesized virtual pixels.

Original: We discuss these twomodules in Sec.
Revised: We discuss these two modules in Sec.

Original: 3.1 and Sec.
Revised: 3.1 and Sec.

Original: 3.2 respectively, followed by training details in Sec.
Revised: 3.2 respectively, followed by training details in Sec.

Original: 3.1 Scene Representation with Neural Radiance Field
Revised: 3.1 Scene Representation with Neural Radiance Field

Original: To provide high-fidelity novel-view synthesis of virtual cameras, we represent the scene with a neuralradiance field (NeRF) [6].
Revised: To provide high-fidelity novel-view synthesis of virtual cameras, we represent the scene with a neural radiance field (NeRF) [6].

Original: For our purposes, a key feature of NeRF is that it renders individual rays (pixels)rather than whole images , which enables flexible parameterization of rendering at inference time, includingcamera models that are non-physical (e.g., orthographic cameras) and not provided in the training set.
Revised: For our purposes, a key feature of NeRF is that it renders individual rays (pixels) rather than whole images , which enables flexible parameterization of rendering at inference time, including camera models that are non-physical (e.g., orthographic cameras) and not provided in the training set.

Original: To render a pixel, NeRF casts a ray r ( t )
Revised: To render a pixel, NeRF casts a ray r ( t )

Original: =
Revised: =

Original: o + t d from some origin o along the direction d passing throughthat pixel on an image plane.
Revised: o + t d from some origin o along the direction d passing through that pixel on an image plane.

Original: In particular, these rays are casted into a field F Θ whose input is a 3D location x =( x,y,z ) and unit-norm viewing direction d , and whose output is an emitted color c =( r,g,b ) and volumedensity σ .
Revised: In particular, these rays are casted into a field F Θ whose input is a 3D location x =( x,y,z ) and unit-norm viewing direction d , and whose output is an emitted color c =( r,g,b ) and volume density σ .

Original: Along each ray, K discrete points { x k
Revised: Along each ray, K discrete points { x k

Original: =
Revised: =

Original: r
Revised: r

Original: ( t k ) }
Revised: ( t k ) }

Original: Kk =1 are sampled for use as input to F Θ , whichoutputs a set of densities and colors { σ k , c k } Kk [-=
-] = { F Θ ( x k , d ) } Kk =1 .
Revised: Kk =1 are sampled for use as input to F Θ , which outputs a set of densities and colors { σ k , c k } Kk [-=
-]=1 = { F Θ ( x k , d ) } Kk =1 .

Original: Volume rendering [62] with a numerical quadrature approximation [63] is performed using these values to produce the color ˆ C ( r ) of that pixel: 
Revised: Volume rendering [62] with a numerical quadrature approximation [63] is performed using these values to produce the color ˆ C ( r ) of that pixel: ˆ C ( r )=

Original: [Equation]
Revised: K (cid:88) k =

Revised: T k (cid:18)

Original: [Equation]
Revised: [Equation]

Original: ) where T k represents the probability that the ray successfully transmits to point r ( t k ) .
Revised: where T k represents the probability that the ray successfully transmits to point r ( t k ) .

Original: At the beginningof each pick-and-place, our system takes multi-view posed RGB images as input and optimizes Θ byminimizing a photometric loss L 
Revised: At the beginning of each pick-and-place, our system takes multi-view posed RGB images as input and optimizes Θ by minimizing a photometric loss L photo =

Original: [Equation]
Revised: (cid:80) r ∈ R ||

Original:  || 22 , using some sampled set of rays r ∈ R ,where C ( r ) is the observed RGB value of the pixel corresponding to ray r in an input image.
Revised: ˆ C ( r ) − C ( r ) || 22 , using some sampled set of rays r ∈ R , where C ( r ) is the observed RGB value of the pixel corresponding to ray r in an input image.

Original: In practice,we use instant-NGP [64] to accelerate NeRF training and inference.
Revised: In practice, we use instant-NGP [64] to accelerate NeRF training and inference.

Original: Orthographic Ray Casting.
Revised: Orthographic Ray Casting.

Original: In Fig.
Revised: In Fig.

Original: 2 we illustrate the difference between perspective and orthographiccameras.
Revised: 3 we illustrate the difference between perspective and orthographic cameras.

Original: Though renderings from a NeRF F Θ are highly realistic, the perspective ray casting procedureused by default in NeRF’s volume rendering, which we visualize in Fig. 2(b), may cause scene content toappear distorted or scaled depending on the viewing angle and the camera’s field of view — more distantobjects will appear smaller in the image plane.
Revised: Though renderings from a NeRF F Θ are highly realistic, the perspective ray casting procedure used by default in NeRF’s volume rendering, which we visualize in Fig. 3(b), may cause scene content to appear distorted or scaled depending on the viewing angle and the camera’s field of view — more distant objects will appear smaller in the image plane.

Original: Specifically, given a pixel coordinate ( u,v ) and camerapose ( R , t ) , NeRF forms a ray r =( o , d ) using the perspective camera model:
Revised: Specifically, given a pixel coordinate ( u,v ) and camera pose ( R , t ) , NeRF forms a ray r =( o , d ) using the perspective camera model:

Original: [Equation]
Revised: [Equation]

Original: This model is a reasonable proxy for the geometry of most consumer RGB cameras, hence its use by
Revised: This model is a reasonable proxy for the geometry of most consumer RGB cameras, hence its use by

Original: NeRF during training and evaluation.
Revised: NeRF during training and evaluation.

Original: However, the distortion and scaling effects caused by perspectiveray casting degrades the performance of the downstream optimization procedure that takes as input thesynthesized images rendered by NeRF, as we will demonstrate in our results (Sec. 4.1).
Revised: However, the distortion and scaling effects caused by perspective ray casting degrades the performance of the downstream optimization procedure that takes as input the synthesized images rendered by NeRF, as we will demonstrate in our results (Sec. 3.4).

Original: To address thisissue, we modify the rendering procedure of NeRF after it is optimized by replacing perspective ray castingwith orthographic ray casting:
Revised: To address this issue, we modify the rendering procedure of NeRF after it is optimized by replacing perspective ray casting with orthographic ray casting:

Original: [Equation]
Revised: [Equation]

Revised: To our knowledge, our work is the first to demonstrate that when a NeRF model is trained on perspective cameras, it may directly be utilized to render orthographic images.

Revised: Such a result is both non-obvious and surprising because rays are now rendered using a different volume rendering procedure from the one used during training time, and the orthographic rendering process essentially corresponds to an out-of-distribution rendering test on the NeRF model.

Original: We visualize this procedure in Fig. 2(c).
Revised: We visualize this procedure in Fig. 3(c).

Original: Orthographic ray casting marches parallel rays into the scene, sothat each rendered pixel represents a parallel window of 3D space.
Revised: Orthographic ray casting marches parallel rays into the scene, so that each rendered pixel represents a parallel window of 3D space.

Original: This property removes the dependencebetween an object’s appearance and its distance to the camera: an object looks the same if it is either far or nearby.
Revised: This property removes the dependence between an object’s appearance and its distance to the camera: an object looks the same if it is either far or nearby.

Original: Further, as all rays for a given camera rotation R are parallel, this provides equivariance to thein-plane camera center c x ,c y .
Revised: Further, as all rays for a given camera rotation R are parallel, this provides equivariance to the in-plane camera center c x ,c y .

Original: These attributes facilitate downstream learning to be equivariant to objects’3D locations and thereby encourages generalization.
Revised: These attributes facilitate downstream learning to be equivariant to objects’ 3D locations and thereby encourages generalization.

Original: As open-source instant-NGP [64] did not supportorthographic projection, we implement this ourselves as a CUDA kernel (see Supp.).
Revised: As open-source instant-NGP [64] did not support orthographic projection, we implement this ourselves as a CUDA kernel (see Supp.).

Original: Our decision of choosing an orthographic view of the scene draws inspiration from previous works thathave used single-view orthographic scene representations [16, 9], but critically differs in the followingtwo aspects: (a) we create orthographic scene representations by casting rays into a radiance field ratherthan by point-cloud reprojection, thereby significantly reducing image artifacts (see Supp.); (b) we relyon multi-view RGB images to recover scene geometry, instead of depth sensors.
Revised: Our decision of choosing an orthographic view of the scene draws inspiration from previous works that have used single-view orthographic scene representations [16, 9], but critically differs in the following two aspects: (a) we create orthographic scene representations by casting rays into a radiance field rather than by point-cloud reprojection, thereby significantly reducing image artifacts (see Supp.); (b) we rely on multi-view RGB images to recover scene geometry, instead of depth sensors.

Original: 3.2 Policy Representation: Affordance Raycasts in a Radiance Field
Revised: 3.2 Policy Representation: Affordance Raycasts in a Radiance Field

Original: To address SE (3) -parameterized actions, we formulate action selection as an optimization problem oversynthesized novel-view pixels and their affordances.
Revised: To address SE (3) -parameterized actions, we formulate action selection as an optimization problem over synthesized novel-view pixels and their affordances.

Original: Using our NeRF-based scene representation, wedensely sample V camera poses around the workspace and render images ˆ I v t = F Θ ( T v t ) for each pose,∀ v t = 0 , 1 , ··· ,V .
Revised: Using our NeRF-based scene representation, we densely sample V camera poses around the workspace and render images ˆ I v t = F Θ ( T v t ) for each pose, ∀ v t = 0 , 1 , ··· ,V .

Original: One valid approach is to search for actions directly in the space of camera poses forthe best T v t , but orders of magnitude computation may be saved by instead considering actions thatcorrespond to each pixel within each image, and sharing computation between all pixels in the image(e.g., by processing each image with just a single pass through a ConvNet).
Revised: One valid approach is to search for actions directly in the space of camera poses for the best T v t , but orders of magnitude computation may be saved by instead considering actions that correspond to each pixel within each image, and sharing computation between all pixels in the image (e.g., by processing each image with just a single pass through a ConvNet).

Original: This (i) extends the paradigmof pixel-wise affordances [7] into full 6-DOF, novel-view-enabled action spaces, and (ii) alleviates thesearch over poses due to translational equivariance provided by orthographic rendering (Sec. 3.1).
Revised: This (i) extends the paradigm of pixel-wise affordances [7] into full 6-DOF, novel-view-enabled action spaces, and (ii) alleviates the search over poses due to translational equivariance provided by orthographic rendering (Sec. 3.1).

Original: Accordingly, we formulate each pixel in each synthesized view as parameterizing a robot action, andwe learn a dense action-value function E which outputs per-pixel action values of shape
Revised: Accordingly, we formulate each pixel in each synthesized view as parameterizing a robot action, and we learn a dense action-value function E which outputs per-pixel action-values of shape

Original: R H × W givena novel-view image of shape R H
Revised: R H × W given a novel-view image of shape R H

Original: × W × 3 .
Revised: × W × 3 .

Original: Actions are selected by simultaneously searching across all pixelsu in all synthesized views v t :
Revised: Actions are selected by simultaneously searching across all pixels u in all synthesized views v t :

Original: [Equation]
Revised: [Equation]

Original: where the pixel u ∗ t and the associated estimated depth
Revised: where the pixel u ∗ t and the associated estimated depth

Original: d ( u ∗ t ) from NeRF are used to determine the 3Dtranslation, and the orientation of T v ∗ t is used to determine the 3D rotation of the predicted action.
Revised: d ( u ∗ t ) from NeRF are used to determine the 3D translation, and the orientation of T v ∗ t is used to determine the 3D rotation of the predicted action.

Original: Ourapproach employs  multiple strategies for equivariance: 3D translational equivariance is in part enabledby orthographic raycasting and synergizes well with translationally-equivariant dense model architecturesfor E such as ConvNets [65, 66], meanwhile 3D rotational equivariance is also encouraged, as synthesizedrotated views can densely cover novel orientations of objects.
Revised: Our approach employs a single ConvNet that is shared across all views and use multiple strategies for equivariance: 3D translational equivariance is in part enabled by orthographic raycasting and synergizes well with translationally-equivariant dense model architectures for E such as ConvNets [65, 66], meanwhile 3D rotational equivariance is also encouraged, as synthesized rotated views can densely cover novel orientations of objects.

Original: While the formulation above may be used to predict the picking pose T pick and the placing pose T placeindependently, intuitively the prediction of T pick affects the prediction of T place due to the latter’s geometricdependence on the former.
Revised: While the formulation above may be used to predict the picking pose T pick and the placing pose T place independently, intuitively the prediction of T pick affects the prediction of T place due to the latter’s geometric dependence on the former.

Original: We therefore decompose the action-value function into (i) picking and (ii)pick-conditioned placing, similar to prior work [9]:
Revised: We therefore decompose the action-value function into (i) picking and (ii) pick-conditioned placing, similar to prior work [9]:

Original: [Equation]
Revised: [Equation]

Original: where E place uses the Transport operation from Zeng et al.
Revised: where E place uses the Transport operation from Zeng et al.

Original: [9] to convolve the feature map of ˆ I v ∗ pick aroundu ∗ pick with the feature maps of { ˆ I v place } Vv place =1 for action-value prediction.
Revised: [9] to convolve the feature map of ˆ I v ∗ pick around u ∗ pick with the feature maps of { ˆ I v place } V v place =1 for action-value prediction.

Original: We refer readers to [9] for detailson this coupling.
Revised: We refer readers to [9] for details on this coupling.

Original: 3.3 Training
Revised: 3.3 Training

Original: We train the action-value function as an energy-based model (EBM) [67, 68, 69].
Revised: We train the action-value function with imitation learning.

Original: For each expertdemonstration, we construct a tuple D
Revised: For each expert demonstration, we construct a tuple D

Original: = { ˆ I v ∗ pick , u ∗ pick , ˆ I v ∗ place , u ∗ place } , where ˆ I v ∗ pick
Revised: = { ˆ I v ∗ pick , u ∗ pick , ˆ I v ∗ place , u ∗ place } , where ˆ I v ∗ pick

Original: and ˆ I v ∗ place are the synthesizedimages whose viewing directions are aligned with the end-effector’s rotations; u ∗ pick and u ∗ place are thebest pixels in those views annotated by experts.We use a Monte-Carlo log-likelihood training objective [Figure] unseenobjects
Revised: and ˆ I v ∗ place are the synthesized images whose viewing directions are aligned with the end-effector’s rotations; u ∗ pick and u ∗ place are the best pixels in those views annotated by experts.

Original: [Figure]

Original: ✓ ✗ ✗ ✗ ✗ [Figure] ✗ ✓ ✓ ✗ ✗ hanging-disks ✓ ✗ ✗ ✓ ✗ stacking-objects ✓ ✗ ✗ ✗ ✓ [70], where we draw pixels { ˆ u j | ˆ u j ̸ = u ∗ pick , ˆ u j ̸ = u ∗ place } N neg j =1 from randomly synthesized images ˆ
Revised: We draw pixels { ˆ u j | ˆ u j ̸ = u ∗ pick , ˆ u j ̸ = u ∗ place } N neg j =1 from randomly synthesized images ˆ

Original: I neg asnegative samples.
Revised: I neg as negative samples.

Original: For brevity, we omit the subscript for pick and place and present the loss function thatis used to train both action-value functions:
Revised: For brevity, we omit the subscript for pick and place and present the loss function that is used to train both action-value functions:

Original: [Equation]
Revised: [Equation]

Original: e − E θ (ˆ I, u ) [Equation] A key innovation of our objective function compared to previous works [9, 10, 36, 71] is the inclusionof negative samples { ˆ u j } N neg j =1 from imagined views ˆ I neg .
Revised: A key innovation of our objective function compared to previous works [9, 10, 36, 67] is the inclusion of negative samples { ˆ u j } N neg j =1 from imagined views ˆ I neg .

Original: We study the effects of ablating negative samplesin Sec. 4.1 and show that they are essential for successfully training action-value functions.
Revised: We study the effects of ablating negative samples in Sec. 3.4 and show that they are essential for successfully training action-value functions.

Original: Results
Revised: In practice, we batch the image that contains the positive pixel and other synthesized views for the forward pass. Every pixel in these images are treated as samples to compute the loss.

Original: We execute experiments in both simulation and real-world settings to evaluate the proposed method acrossvarious tasks.
Revised: We execute experiments in both simulation and real-world settings to evaluate the proposed method across various tasks.

Original: 4.1 Simulation Experiments
Revised: 3.4 Simulation Experiments

Original: Environment.
Revised: Environment.

Original: We propose four new 6-DoF tasks based on Ravens [9] and use them as the benchmark.
Revised: We propose four new 6-DoF tasks based on Ravens [9] and use them as the benchmark.

Original: We show qualitative examples of these tasks in Fig.
Revised: We show qualitative examples of these tasks in Fig.

Original: 3 and summarize their associated challenges in Table 1.
Revised: 4 and summarize their associated challenges in the supplementary materials.

Revised: Notably, place-red-in-green is a relatively cluttered environment where 5distractor objects are randomly spawned and placed; hanging-disks and stacking-objects require the policy to generalize to novel objects that are not seen during training.

Original: All simulated experiments are conducted in PyBullet [72] using a Universal Robot UR5e with a suctiongripper.
Revised: All simulated experiments are conducted in PyBullet [68] using a Universal Robot UR5e with a suction gripper.

Original: The input observations for MIRA are 30 RGB images from different cameras pointing toward thecenter.
Revised: The input observations for MIRA are 30 RGB images from different cameras pointing toward the center.

Original: For all the baselines, we additionally supply the corresponding noiseless depth images.
Revised: For all the baselines, we additionally supply the corresponding noiseless depth images.

Original: Each imagehas a resolution of 640 × 480 .
Revised: Each image has a resolution of 640 × 480 .

Original: The camera has focal length f
Revised: The camera has focal length f

Original: =450 and camera center ( c x ,c y )=(320 , 240) .
Revised: = 450 and camera center ( c x ,c y ) =

Revised: Demonstrations are collected with motion planner that can access the ground-truth states of objects.

Original: Evaluation.
Revised: Evaluation.

Original: For each task, we perform evaluations under two settings: in-distribution configuresobjects with random rotations ( θ x ,θ y ∈ [ − π 6 , π 6 ] ,θ z ∈ [ − π,π ] ).
Revised: For each task, we perform evaluations under two settings: in-distribution configures objects with random rotations ( θ x ,θ y ∈ [ − π 6 , π 6 ] ,θ z ∈ [ − π,π ] ).

Original: This is also the distribution we usedto construct the training set.
Revised: This is also the distribution we used to construct the training set.

Original: out-of-distribution instead configures objects with random rotations( θ x ,θ y ∈ [ − π 4 , − π 6 ] ∪ [ π 6 , π 4 ] ,θ z ∈ [ − π,π ]) .
Revised: out-of-distribution instead configures objects with random rotations ( θ x ,θ y ∈ [ − π 4 , − π 6 ] ∪ [ π 6 , π 4 ] ,θ z ∈ [ − π,π ]) .

Original: We note that these rotations are outside the training distributionand also correspond to larger out-of-plane rotations.
Revised: We note that these rotations are outside the training distribution and also correspond to larger out-of-plane rotations.

Original: Thus, this setting requires stronger generalization.
Revised: Thus, this setting requires stronger generalization.

Original: We use a binary score (0 for failure and 1 for success) and report results on 100 evaluation runs for agentstrained with n =1 , 10 , 100 demonstrations.
Revised: We use a binary score (0 for failure and 1 for success) and report results on 100 evaluation runs for agents trained with n =1 , 10 , 100 demonstrations.

Original: Baseline Methods.
Revised: Baseline Methods.

Original: Although our method only requires RGB images, we benchmark against publishedbaselines that additionally require depth images as inputs.
Revised: Although our method only requires RGB images, we benchmark against published baselines that additionally require depth images as inputs.

Original: Form2Fit [14] predicts the placing action byestimating dense descriptors of the scene for geometric matching.
Revised: Form2Fit [14] predicts the placing action by estimating dense descriptors of the scene for geometric matching.

Original: Transporter-
Revised: Transporter-

Original: SE (2) and Transporter- SE (3)are both introduced in Zeng et al.
Revised: SE (2) and Transporter- SE (3) are both introduced in Zeng et al.

Original: Although Transporter- SE (2) is not designed to solve manipulationtasks that require 6-DoF actions, its inclusion helps indicate what level of task success can be achievedon the shown tasks by simply ignoring out-of-plane rotations.
Revised: Although Transporter- SE (2) is not designed to solve manipulation tasks that require 6-DoF actions, its inclusion helps indicate what level of task success can be achieved on the shown tasks by simply ignoring out-of-plane rotations.

Original: Transporter-
Revised: Transporter-

Original: SE (3) predicts 6-DoF actions by first using Transporter- SE (2) to estimate SE (2) actions, and then feeding them into a regression modelto predict the remaining rotational ( r x ,r y ) and translational (z-height) degrees of freedom.
Revised: SE (3) predicts 6-DoF actions by first using Transporter- SE (2) to estimate SE (2) actions, and then feeding them into a regression model to predict the remaining rotational ( r x ,r y ) and translational (z-height) degrees of freedom.

Original: Additionally,we benchmark against a baseline, GT-State MLP, that assumes perfect object poses.
Revised: Additionally, we benchmark against a baseline, GT-State MLP, that assumes perfect object poses.

Original: It takes ground truthstate (object poses) as inputs and trains an MLP to regress two SE (3) poses for Tpick and T
Revised: It takes ground truth state (object poses) as inputs and trains an MLP to regress two SE (3) poses for T

Original: [Table]
Revised: [Table]

Revised: and T

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  [Table] [Table]
Revised: [Figure] [Table] [Figure]

Original: [Table] [Table] [Table]
Revised: [Table] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 4 shows the average scores of all methods trained on different numbers ofdemonstrations.
Revised: 5 shows the average scores of all methods trained on different numbers of demonstrations

Original:  GT-State MLP fails completely; Form2Fit cannot achieve 40% successrate under any setting; Both Transporter- SE (2) and Transporter- SE (3) are able to achieve∼ 70% success rate when the object poses are sampled from the training distribution, butthe success rate drops to ∼ 30% when the object poses are outside the training distribution.
Revised: [Table] GT-State MLP fails completely; Form2Fit cannot achieve 40% success rate under any setting; Both Transporter- SE (2) and Transporter- SE (3) are able to achieve ∼ 70% success rate when the object poses are sampled from the training distribution, but the success rate drops to ∼ 30% when the object poses are outside the training distribution.

Original: MIRA outperforms all baselinesby a large margin when thereare enough demonstrations ofthe task.
Revised: MIRA outperforms all baselines by a large margin when there are enough demonstrations of the task.

Original: Its success rate is∼ 90% under the setting ofin-distribution and ∼ 80% underthe setting of out-of-distribution .
Revised: Its success rate is ∼ 90% under the setting of in-distribution and ∼ 80% under the setting of out-of-distribution .

Original: The performance improvementover baselines demonstrates itsgeneralization ability thanks tothe action optimization processwith novel view synthesis andaffordance prediction in the loop.
Revised: The performance improvement over baselines demonstrates its generalization ability thanks to the action optimization process with novel view synthesis and affordance prediction in the loop.

Original: Interestingly, we found that MIRA sometimes performs worse thanbaselines when only 1 demonstration is supported.
Revised: Interestingly, we found that MIRA sometimes performs worse than baselines when only 1 demonstration is supported.

Original: We hypothesize that this is because our action-valuefunction needs more data to understand the subtle differences between images rendered from differentviews in order to select the best view.
Revised: We hypothesize that this is because our action-value function needs more data to understand the subtle differences between images rendered from different views in order to select the best view.

Original: We show the full quantitative results in Table 2.
Revised: We show the full quantitative results in Table 1.

Original: Ablation studies.
Revised: Ablation studies.

Original: To understand the importance of different components within our framework, webenchmark against two variants: (i) ours w/ perspective ray casting, and (ii) ours w/o multi-view negativesamples in Eq.
Revised: To understand the importance of different components within our framework, we benchmark against two variants: (i) ours w/ perspective ray casting, and (ii) ours w/o multi-view negative samples in Eq.

Original: We show the quantitative results in Table 3.
Revised: We show the quantitative results in Table 2.

Original: We find that ours w/ perspective ray castingfail to learn the action-value functions because the perspective images (a) contain distorted appearancesof the objects that are challenging for CNNs to comprehend and (b) the groudtruth picking or placinglocations may be occluded by the robot arms.
Revised: We find that ours w/ perspective ray casting fail to learn the action-value functions because the perspective images (a) contain distorted appearances of the objects that are challenging for CNNs to comprehend and (b) the groudtruth picking or placing locations may be occluded by the robot arms.

Original: We visualize these challenges in the supplementary materials.
Revised: We visualize these challenges in the supplementary materials.

Original: Our method with orthographic ray casting circumvents both challenges by controlling the near/far planes toignore occlusions without worrying about the distortion and scaling.
Revised: Our method with orthographic ray casting circumvents both challenges by controlling the near/far planes to ignore occlusions without worrying about the distortion and scaling.

Original: Ours w/o multi-view negative samples also fails to learn reliable action-value functions, and this may be due to a distribution shift between trainingand test: during training, it has only been supervised to choose the best pixel given an image.
Revised: Ours w/o multi-view negative samples also fails to learn reliable action-value functions, and this may be due to a distribution shift between training and test: during training, it has only been supervised to choose the best pixel given an image.

Original: However, it istasked to both select the best pixel and best view during the test time.
Revised: However, it is tasked to both select the best pixel and best view during the test time.

Original: [Figure]
Revised: [Table]

Original: 4.2 Real-world Experiments
Revised: 3.5 Real-world Experiments

Original: We validate our framework with a kitting task in the real world and show qualitative results in Fig.
Revised: We validate our framework with three kitting tasks in the real world and show qualitative results in Fig.

Revised: Additional qualitative results and video can be found in the supplementary material.

Original: Oursystem consists of a UR5 arm, a customized suction gripper, and a wrist-mounted camera.
Revised: Our system consists of a UR5 arm, a customized suction gripper, and a wrist-mounted camera.

Original: We show that ourmethod can successfully  pick up a stainless steel ice sphere and place it into 10+ different cups configuredwith random translations and out-of-plane rotations. 
Revised: We show that our method can successfully (i) pick up floss cases and pack them into transparent containers, (ii) pick up metal cubes and insert them into the cases, and (iii) pick up a stainless steel ice sphere and place it into 10+ different cups configured with random translations and out-of-plane rotations. See Fig.

Original: This task is challenging because (i) it includes objectswith reflective or transparent materials, which makes this task not amenable to existing works that requiredepth sensors [9, 12, 14], and (ii) it requires out-of-plane action reasoning.
Revised: 2 for qualitative results. These tasks are challenging because (i) they include objects with reflective or transparent materials, which makes these tasks not amenable to existing works that require depth sensors [9, 12, 14], and (ii) they require out-of-plane action reasoning.

Original: The action-value functions aretrained with 20 demonstrations using these cups.
Revised: The action-value functions are trained with 20 demonstrations using these cups.

Revised: Demonstrations are supplied by humans who teleoperate the robot through a customized user interface.

Original: At the beginning of each pick-and-place, our systemgathers 30 1280 × 720 RGB images of the scene with the wrist-mounted camera.
Revised: At the beginning of each pick-and-place, our system gathers 30 1280 × 720 RGB images of the scene with the wrist-mounted camera.

Revised: These 30 locations are sampled from a circular path on the table and the robot moves the camera to each of them sequentially through inverse kinematics.

Original: Each image’s camerapose is derived from the robotic manipulator’s end-effector pose and a calibrated transformation betweenthe end-effector and the camera.
Revised: Each image’s camera pose is derived from the robotic manipulator’s end-effector pose and a calibrated transformation between the end-effector and the camera.

Original: This data collection procedure is advantageous as industrial roboticmanipulators feature sub-millimeter repeatability, which provides accurate camera poses for building NeRF.
Revised: This data collection procedure is advantageous as industrial robotic manipulators feature sub-millimeter repeatability, which provides accurate camera poses for building NeRF.

Original: In practice, we search through V =121 virtual views that uniformly cover the workspace and predict theiraffordances for optimizing actions.
Revised: In practice, we search through V =121 virtual views that uniformly cover the workspace and predict their affordances for optimizing actions.

Original: The optimization process currently takes around 2 seconds using asingle NVIDIA RTX 2080 Ti GPU.
Revised: The optimization process currently takes around 2 seconds using a single NVIDIA RTX 2080 Ti GPU.

Original: This step can be straightforwardly accelerated by parallelizing thecomputations with multiple GPUs.
Revised: This step can be straightforwardly accelerated by parallelizing the computations with multiple GPUs.

Original:  Limitations And Conclusion
Revised:  Limitations And Conclusion

Original: In terms of limitations, our system currently requires training a NeRF of the scene for each step of themanipulation.
Revised: In terms of limitations, our system currently requires training a NeRF of the scene for each step of the manipulation.

Original: An instant-NGP [64] requires approximately 10 seconds to converge using a single NVIDIA
Revised: An instant-NGP [64] requires approximately 10 seconds to converge using a single NVIDIA

Original: RTX 2080 Ti GPU, and moving the robot arms around to collect 30 multi-view RGB images of the scenetakes nearly 1 minute.
Revised: RTX 2080 Ti GPU, and moving the robot arms around to collect 30 multi-view RGB images of the scene takes nearly 1 minute.

Revised: This poses challenges to apply MIRA to tasks that require real-time visio-motor control.

Original: We believe observing the scene with multiple mounted cameras or learning a priorover instant-NGP could drastically reduce the runtime.
Revised: We believe observing the scene with multiple mounted cameras or learning a prior over instantNGP could drastically reduce the runtime.

Original: In the future, we plan to explore the usage of mentalimagery for other robotics applications such as navigation and mobile manipulation.
Revised: In the future, we plan to explore the usage of mental imagery for other robotics applications such as navigation and mobile manipulation.