NeRF-GTO: Using a Neural Radiance Field to Grasp Transparent Objects
Anonymous s)
AfﬁliationAddress email
Abstract: The ability to grasp and manipulate transparent objects is a major challenge for robots.
Existing depth cameras have difﬁculty detecting, localizing, andinferring the geometry of such objects.
We propose using neural radiance ﬁelds(NeRF) to detect, localize, and infer the geometry of transparent objects withsufﬁcient accuracy to ﬁnd and grasp them securely.
We leverage NeRF’s viewindependent learned density, place lights to increase specular reﬂections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net graspplanner.
We show how additional lights create specular reﬂections that improvethe quality of the depth map, and test a setup for a robot workcell equipped withan array of cameras to perform transparent object manipulation.
We also createsynthetic and real datasets of transparent objects in real-world settings, includingsingulated objects, cluttered tables, and the top rack of a dishwasher.
In each setting we show that NeRF and Dex-Net are able to reliably compute robust graspson transparent objects, achieving 90 % and 100 % grasp success rates in physicalexperiments on an ABB YuMi, on objects where baseline methods fail.
See supplementary material and https://sites.google.com/view/nerf-gto/homefor code, video, and datasets.
Introduction
Transparent objects are common in homes, restaurants, retail packaging, labs, gift shops, hospitals,and industrial warehouses.
Effectively automating robotic manipulation of transparent objects couldhave a broad impact, from helping in everyday tasks and performing tasks in hazardous environments.
Existing depth cameras assume that surfaces of observed objects reﬂect light uniformly inall directions, but this assumption does not hold for transparent objects as their appearance variessigniﬁcantly under different view directions and illumination conditions due to reﬂection and refraction.
In this paper, we propose and demonstrate NeRF-GTO , a new method to sense the geometry oftransparent objects and allow for robots to interact with them—potentially enabling automation ofnew tasks.
NeRF-GTO uses a Neural Radiance Fields (NeRF) as part of a pipeline (Fig. 1, right) to compute andexecute robots grasps on transparent objects.
While NeRF was originally proposed as an alternativefor explicit volumetric representations to render novel views of complex scenes [1], it can alsoreconstruct the geometry of the scene.
In particular, due to the view-dependent nature of the NeRFmodel, it can learn to accurately represent the geometry associated with transparency.
The only inputrequirement to train a NeRF model is a set of images taken from a camera with known intrinsics(e.g. , focal length, distortion) and extrinsics (position and orientation in the world).
While theintrinsics can be determined from calibration techniques, and/or from the camera itself, determiningthe extrinsics is often a challenge [2, 3]; however, robots in a ﬁxed workcell, or using a cameraattached to a manipulator arm with accurate encoders, can readily determine camera extrinsics.
Thismakes NeRF a particularly good match for robot manipulators.
In experiments, we show qualitatively and quantitatively that NeRF-based grasp-planning canachieve high accuracy on NeRF models trained from photo-realistic synthetic images and from realimages, and achieve 90 % or better grasp success rates on real objects. DatasetGeneration
[Figure]
The contributions of this paper are: (1) Integration of NeRF with robot grasp planning, (2) A transparency-aware depth rendering method for NeRF, (3) Experiments on synthetic and real imagesshowing NeRF with Dex-Net generates high-quality grasps, (4) Synthetic and real image datasetswith transparent objects for training NeRF models.
Related Work
Detecting Transparent Objects For robots to interact with transparent objects, they must ﬁrst beable to detect them.
The most recent approaches detecting and recognizing transparent objects aredata-driven.
Lai et al.
[4], then Khaing et al.
[5], proposed using a Convolutional Neural Network(CNN) to detect transparent objects in RGB images.
Recently, Xie et al.
[6] developed a transformerbased pipeline [7] used for transparent object segmentation.
Other methods rely on deep-learningmodels to predict the object pose.
Phillips et al.
[8] trained a random forest to detect the contoursof transparent objects for the purpose of pose estimation and shape recovery.
Xu et al.
[9] proposeda two-stage method for estimating the 6-degrees-of-freedom (DOF) pose of a transparent objectwith a single RGBD image by replacing the noisy depth values with estimated values and traininga DenseFusion-like network structure [10] to predict the object’s 6-DOF pose.
Sajjan et al.
[11]extend this and incorporate a neural network trained for 3D pose estimation of transparent objectsin a robotic picking pipeline, while Zhou et al.
[12, 13] train a grasp planner directly on raw imagesfrom a light-ﬁeld camera.
Zhu et al.
[14] used an implicit function to complete missing depth givennoisy RGBD observation of transparent objects.
However, these data-driven methods rely on largeannotated datasets that are hard to curate, whereas NeRF-GTO does not require any prior dataset.
Neural Radiance Fields Recently, implicit neural representations have led to signiﬁcant progressin 3D object shape representation [15, 16, 17] and encoding the geometry and appearance of 3Dscenes [18, 1].
Mildenhall et al.
[1] presented Neural Radiance Fields (NeRF), a neural networkwhose input is a 3D coordinate with an associated view direction, and output is the volume densityand view-dependent emitted radiance at that coordinate.
Due to its view-dependent emitted radianceprediction, NeRF can be used to represent non-Lambertian effects such as specularities and reﬂections, and therefore capture the geometry of transparent objects.
However, NeRF is slow to trainand has low data efﬁciency.
Yu et al.
[19] proposed Plenoctrees , mapping coordinates to spherical harmonic coefﬁcients, shifting the view-dependency from the input to the output.
In addition, Plenoctrees pre-samples the model into a sparse octree structure, achieving a signiﬁcant speedup intraining over NeRF.
Deng et al.
[20] proposed JaxNeRF, an efﬁcient JAX implementation of NeRFthat was able to reduce the training time of a NeRF model from over a day to several hours.
Deng etal.
[21] add depth supervision to train NeRF 2 to 6 × faster given fewer training views.
In this work,we propose to use NeRF to recover the geometry of transparent objects for the purpose of roboticmanipulation.
Robotic Grasping Traditional robot grasping methods analyze the object shape to identify successful grasp poses [22, 23, 24].
Data-driven approaches learn a prior using labeled data [25, 26]or through self-supervision over many trials in a simulated or physical environment [27, 28] andgeneralize to grasping novel objects with unknown geometry.
Both approaches rely on RGB anddepth sensors to generate a sufﬁciently accurate observation of the target object surface, such asdepth maps [29, 30, 31], point clouds [32, 33, 34, 9], octrees [35], or a truncated signed distancefunction (TSDF) [36, 37] from which it can compute the grasp pose.
While various grasp-planningmethods use different input geometry to compute grasps, in this paper we propose a method to render a high-quality depth map from a NeRF model to then pass to Dex-Net [29] to compute a grasp.
While standard depth cameras have gaps in their depth information that needs to be processed outwith hole-ﬁlling techniques, the depth map rendering from NeRF is directly usable.
We observe thatother grasp-planning techniques may similarly be able to plan grasps from NeRF models as well.
Problem Statement
Given an environment containing rigid transparent objects, compute a frame for a robot gripper thatwill result in a stable grasp of a transparent object.
We assume the environment has an array ofcameras at ﬁxed known locations, or that the robot can manipulate a camera (e.g., wrist-mounted)to capture multiple images of the scene.
Method
In this section, we provide a brief background on NeRF, then describe recovering geometry of transparent objects, integrating with grasp analysis, and improving performance with additional lights.
4.1 Preliminary: Training NeRF
NeRF [1] learns a neural scene representation that maps a 5D coordinate containing a spatial location ( x, y, z ) and viewing direction ( θ, φ ) to the volume density σ and RGB color c .
Training NeRF’s multilayer perceptron (MLP) requires multi-view RGB images of a static scene with theircorresponding camera poses and intrinsic parameters.
The expected color C ( r ) of the camera rayr = o + t d between near and far scene bounds t n and t f is:
[Equation]
where T ( t ) = exp (cid:16) − (cid:82) t t n σ ( r ( s ))
ds (cid:17) is the probability that the camera ray travels from near boundt n to point t without hitting any surface.
NeRF approximates the expected color ˆ C ( r ) as:
[Equation]
where T i = exp (cid:16) − (cid:80) i − 1 j =1 σ j δ j (cid:17) and δ i
=
t i +1 − t
i is the distance between consecutive samples onthe ray r .
During training, NeRF minimizes the error between the rendered and ground truth rays’colors using gradient descent.
We observe that NeRF does not directly support transparent object effects—it casts a single rayper source image pixel without reﬂection, splitting, or bouncing.
The incorporation of the viewingdirection in its regression and supervising with view-dependent emitted radiance allows recovery ofnon-Lambertian effects such as reﬂections from a specular surface. However, while RGB color c isview-dependent, the volume density σ is not—meaning NeRF has to learn a non-zero σ to representany color at that spatial location.
Visually, the usual result is that the transparent object shows up asa “ghostly” or “blurry” version of the original object.
When training, a NeRF model learns a density σ of each spatial location.
This density correspondsto the transparency of the point, and serves to help learn how much a spatial location contributes to
[Figure]
[Figure]
the color of a ray cast through it. Although NeRF converts each σ i to an occupancy probability α i =1 − exp( − σ i δ i ) , where δ i is the distance between integration times along the ray, thus implicitlygiving α i an upper bound of 1, it does not place a bound on the raw σ value. We use the raw valueof σ to determine if a point in space is occupied.
4.3 Rendering Depth for Grasp Analysis
We propose using Dex-Net to compute grasps on transparent objects.
Dex-Net computes candidategrasp poses given a depth image of a scene.
To generate a depth image, we consider two candidatereconstructions of depth.
First, we use the same rendering of sampled points along a camera ray that NeRF uses.
This Vanilla NeRF reconstruction ﬁrst converts σ i to an occupancy probability α i .
Itthen applies the transformation w i =
α
i (cid:81) i − 1 j =1 (1 − α j ) .
To render depth at pixel coordinate [ u, v ] ,it computes the sum of sample distances from the camera weighted by the termination probability D [ u, v ] = (cid:80) Ni
=1 w i
δ i .
When applied on transparent objects, however, this results in noisy depthmaps, as shown in Fig.
Instead, we consider a second, transparency-aware method that searches for the ﬁrst sample alongthe ray for which σ i > m where m is a ﬁxed threshold.
The depth is then set to the distance of thatsample δ i .
We explore different values for m , and observe that low values result in a noisy depthmap while high values create holes in the depth map.
In our experiments we set m = 15 (see Fig.
4.4 Improving Reconstruction with Light Placement
For NeRF to learn the geometry of a transparent object, it must be able to “see” it from multiplecamera views.
If the transparent object is not visible from any views, then it will have no effecton the loss function used in training, and thus not be learned.
We thus look for a way to improvevisibility of transparent objects to NeRF.
One property that transparent objects share (e.g., glass, clear plastic) is that they are glossy and thusproduce specular reﬂections when the camera view direction is opposite to the surface normal of theincident direction of light.
To the NeRF model, a specular reﬂection viewed from multiple angleswill appear as a white point on a solid surface—i.e., c
=
[1 , 1 , 1] T and σ > 0 , while from otherangles it will appear as σ ≤ 0 .
As σ is view-independent, NeRF learns a σ between fully opaqueand fully transparent for such points.
By placing additional lights in the scene, we create more angles from which cameras will see specular reﬂections from transparent objects—this results in NeRF learning a model that ﬁlls holes in thescene.
While the number and placement of lights for optimal training is dependent on both the expected object distribution and camera placement, in experiments (Sec. 5.5) we show that increasingfrom 1 light to a 5x5 array of lights improves the quality of the learned geometry.
Experiments
We experiment in both simulation and on a physical ABB YuMi robot. We generate multipledatasets, where each dataset consists of images and associated camera transforms of one static scenecontaining one or more transparent objects.
We train NeRF models using a modiﬁed JaxNeRF [20]implementation on 4 Nvidia V100 GPUs.
We use an existing pre-trained Dex-Net model for graspplanning without modiﬁcation or ﬁne-tuning. We can do this since NeRF models can be renderedto depth maps from arbitrary camera intrinsics and extrinsics, thus we match our NeRF rendering tothe Dex-Net model instead of training a new one.
5.1 Datasets
As existing NeRF datasets do not include transparent objects, and existing transparent-objectgrasping datasets do not include multiple camera angles, we generate new datasets using 3 differentmethods: synthetic, Cannon EOS 60D camera with a Tamron Di II lens with a locked focal length,and an Intel RealSense.
For synthetic datasets, we use Blender 2.92’s physically-based Cycles renderer with path tracingset to 10240 samples per pixel, and max light path bounces set to 1024.
We chose theses settingsby increasing them until renderings were indistinguishable from the previous setting—ﬁnding thatlower settings lead to dark regions and smaller specular reﬂections.
For glass materials, we set theindex of refraction to 1.45 to match physical glass.
We include 8 synthetic datasets of transparentobjects: 2 scenes with clutter: light array and single light; 4 singulated objects from Dex-Net: Pipe Connector, Pawn, Turbine Housing, Mount; and 2 household objects: Wineglass upright and Wineglass on side. As these computationally demanding to render due to the high quality settings,we distribute these as part of the contribution.
For the Cannon EOS and RealSense real-world datasets, we place ArUco markers in the scene toaid in camera pose recovery and take photos around the objects using a ﬁxed ISO, f-stop, and focallength.
We use bundle adjustment from COLMAP [2, 3] to reﬁne the camera poses and intrinsics tohigh accuracy.
We include 8 physical datasets of transparent objects with a variety of camera poses:table with clutter, Dishwasher, Tape Dispenser, Wineglass on side, Flask, Safety Glasses, Bottleupright, Lion Figurine in clutter. The main difﬁculty in generating these datasets is calibration andcomputing high-precision camera poses. The datasets differ from other datasets in their focus on scenes with transparent objects in a graspablesetting, with over 70 camera poses each.
5.2 Synthetic Grasping Experiments
We also test the ability of NeRF-GTO to generate grasps on the synthetic singulated transparent Dex-Net object datasets. For each dataset, we evaluate the grasp in simulation using a wrenchresistance metric measuring the ability of the grasp to resist gravity [38]. Fig.
4 shows images ofthe synthetic objects, NeRF-GTO-generated depth map, and an example sampled grasp for each.
Tomeasure the effect of training time on grasp-success rate, we simulate and record grasps over thecourse of training.
In Fig.
5, we observe that grasp success rate improves with training time, butplateaus between 80 % and 98 % success rate at around 50k to 60k iterations.
This suggests thatthere may be a practical ﬁxed iteration limit to obtain high grasp success rates.
[Figure]
[Figure]
[Figure]
[Figure]
PhoXi
Vanilla NeRF
NeRF-GTO
Tape Dispenser 0/
Wineglass 0/ 0/ 9/
Flask 0/ 1/ 9/
Safety Glasses 0/ 0/
Bottle 0/ 10/ 10/ Lion Figurine 0/ 3/ 10/ We test NeRF-GTO on a scene of a tabletop cluttered with transparent objects. In this experiment,the goal is to grasp a transparent object placed in a stable pose in close proximity to other transparentobjects.
The challenge is twofold: the depth rendering quality should be sufﬁcient for both graspplanning and collision avoidance.
Fig.
1 shows the robot and scene in the upper left, and the overheadimage, depth, and computed grasp inline in the pipeline, and the ﬁnal computed grasp with simulatedexecution is in the upper middle image.
The ﬁnal grasp contact point was accurate to a 2 mmtolerance, suggesting that NeRF-GTO with sufﬁcient images taken from precisely-known cameralocations may be practical in highly clutter environments.
5.3 Physical Grasping Experiments
To test the NeRF-GTO in a physical setup, we place transparent singulated objects in front of an ABB YuMi robot, and have the robot perform the computed grasps. We compare to 2 baselines: (1)
PhoXi , in which a PhoXi camera provides the depth map; and (2) Vanilla NeRF , in which we use theoriginal depth rendering from NeRF. The PhoXi camera is normally able to generate high-precisiondepth maps for non-transparent objects. All methods use the same pre-trained Dex-Net model, andboth Vanilla NeRF and NeRF-GTO use the same NeRF model—the only difference is the depthrendering. We test with 6 objects (Fig. 6), and compute and execute 10 different grasps for each andrecord the success rate.
A grasp is successful if the robot lifts the object.
In Table 1, we see that NeRF-GTO gets 90 % and 100 % success rates for all objects, while the baselines get few successfulgrasps. The PhoXi camera is unable to recover any meaningful geometry which causes Dex-Netpredictions to fail. The Vanilla NeRF depth maps often have unpredictable protrusions that result in
Dex-Net generating unreliable grasps. 5.4 Comparison to RealSense Depth We qualitatively compare the the rendered depth map of the proposed pipeline against a readilyavailable depth sensor on scenes with transparent objects in real-world settings. We select the Intel
[Figure]
[Equation]
RealSense as it is common to robotics applications, readily available, and high-performance.
The RealSense, like most stereo depth cameras, struggles with transparent objects as they are unable tocompute a stereo disparity between pixels from different cameras when the pixels are specular reﬂections or the color of the object behind the transparent object. The RealSense optionally projectsa structured light pattern on the the scene to aid in computing depth from textureless surfaces, however, in experiments we observed no qualitative difference with and without the light pattern emitterenabled. To run this experiment, we use a Canon EOS as described in the main text, and use a
RealSense to take a depth picture of the scene. In this experiment, we observe that the RealSenseis unable to compute the depth of most transparent objects, and often produces regions of unknowndepth (shown in black) where transparent objects are. On the other hand, the proposed pipelineproduces high-quality depth maps with only a few areas of noise. The results of these experimentsis shown in Fig.
5.5 One vs Many Lights
We experiment with different light setups to test the effect of specular reﬂections on the ability of
NeRF to recover geometry of transparent objects.
We create two scenes (Fig. 7), one with a singlebright light source directly above the work surface, and another with an array of 5x5 (25) lights abovethe work surface.
We set the total wattage of the lights in each scene to be the same.
Since mostlights in the multiple light scene are further away from the work surface than the single light source,the scene appears darker, though more evenly illuminated.
The effect of the specular reﬂections isprominent on the lightbulb in the lower part of the image.
In the single light source, there is a singlespecular reﬂection, while in the multiple light scene, the reﬂection of the array of lights is visible.
With the same camera setup for both scenes, we train NeRF models with the same number of iterations.
We show the depth rendering in Fig.
7 and circle a glass and a wineglass on their side.
In thesingle-light source image, the closer surfaces of the glasses are missing, while in the multiple-lightsource depth image, the glasses are nearly fully recovered.
This suggests that additional lights in thescene can help NeRF recover the geometry of transparent objects better. [Figure]
5.6 Workcell Setup
We experiment with a potential setup for a robot workcell in which a grid of overhead camerascaptures views of the cluttered scene so that a robot manipulator arm can then perform tasks withtransparent objects in the workcell.
We propose that a grid of overhead cameras would be practicalto setup and would not obstruct manipulator tasks nor operator interventions.
The objective is to determine how many overhead cameras would be needed to recover a depth map of sufﬁcient accuracyto perform maniplation tasks.
We place a 2 m by 2 m grid of cameras 1 m above the work surface, and have them all point at thecenter of the work surface.
Each camera has the same intrinsics, and are evenly spaced along thegrid.
We experiment with grids having 4, 9, 16, 25, 36, and 49 cameras.
The environment has thesame 5x5 grid of lights as before.
For each camera grid, we train JaxNeRF for 50k iterations andcompare performance.
After training, we observe increasing peak signal to noise ratios (PSNR) and structural similarity(SSIM) scores with increasing number of cameras.
The 2x2 grid of cameras produces a high train-totest ratio for PSNR, likely indicating overﬁtting to training data, and results in a depth map withoutapparent geometry.
This ratio decreases with additional cameras.
The minimum number of camerasfor this proposed setup appears to be around 9 (3x3) as its depth map is usable for grasp planning,while the 5x5 grid shows better PSNR and SSIM and ratio between train and test PSNR, and the 7x7grid is the best.
See Fig.
9 for a visual comparison.
Additionally, we trained 9x9, 11x11, and 13x13grids, observing no statistically signiﬁcant improvement beyond the 7x7 grid.
Conclusion
In this work, we showed that NeRF can recover the geometry of transparent objects with sufﬁcientaccuracy to compute grasps for robot manipulation tasks.
NeRF learns the density of all pointsin space, which corresponds to how much the view-dependent color of each point contributes torays passing through it.
With the key observation that specular reﬂections on transparent objectscause NeRF to learn a non-zero density, we recover the geometry of transparent objects through acombination of additional lights to create specular reﬂections and thresholding to ﬁnd transparentpoints that are visible from some view directions.
With the geometry recovered, we pass it to a graspplanner, and show that the recovered geometry is sufﬁcient to compute a grasp, and accurate enoughto achieve 90 % and 100 % grasp success rates in physical experiments on an ABB YuMi robot.
Wecreated synthetic and real datasets for experiments in transparent geometry recovery, but we believethese datasets may be of interest to researchers interested in extending NeRF capabilities in otherways and thus contribute them as part of this research project.
Finally, to test if NeRF could be usedin a robot workcell, we experimented with grids of cameras facing a worksurface and their ability torecover geometry in potential setup, and showed the increased capabilities and point of diminishingreturn for additional cameras.
In future work, we hope to address one of the main drawbacks of NeRF—the long training timerequired to obtain a NeRF model.
Already many research groups have sped up training time throughimproved implementations, new algorithms and network architectures, pre-conditioned networkweights, focused sampling, and more.
While these approaches apply to general NeRF training,we plan to exploit features speciﬁc to robot scenerios to speed up training, including using depthcamera data as additional training data, manipulator-arm-mounted cameras to inspect regions ofinterest, and visio-spatial foresight to adapt to changes in the environment.