Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
Abstract: The ability to grasp and manipulate transparent objects is a major challenge for robots.
Existing depth cameras have difﬁculty detecting, localizing, and inferring the geometry of such objects.
We propose using neural radiance ﬁelds (NeRF) to detect, localize, and infer the geometry of transparent objects with sufﬁcient accuracy to ﬁnd and grasp them securely.
We leverage NeRF’s viewindependent learned density, place lights to increase specular reﬂections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net grasp planner.
We show how additional lights create specular reﬂections that improve the quality of the depth map, and test a setup for a robot workcell equipped with an array of cameras to perform transparent object manipulation.
We also create synthetic and real datasets of transparent objects in real-world settings, including singulated objects, cluttered tables, and the top rack of a dishwasher.
In each setting we show that NeRF and Dex-Net are able to reliably compute robust grasps on transparent objects, achieving 90 % and 100 % grasp success rates in physical experiments on an ABB YuMi, on objects where baseline methods fail.
See https://sites.google.com/view/dex-nerf for code, video, and datasets.
Introduction
Transparent objects are common in homes, restaurants, retail packaging, labs, gift shops, hospitals and industrial warehouses.
Effectively automating robotic manipulation of transparent objects could have a broad impact, from helping in everyday tasks and performing tasks in hazardous environments.
Existing depth cameras assume that surfaces of observed objects reﬂect light uniformly in all directions, but this assumption does not hold for transparent objects as their appearance varies signiﬁcantly under different view directions and illumination conditions due to reﬂection and refraction properties of transparent materials.
In this paper, we propose and demonstrate Dex-NeRF , a new method to sense the geometry of transparent objects and allow for robots to interact with them.
Dex-NeRF uses a Neural Radiance Fields (NeRF) as part of a pipeline (Fig. 1, right) to compute and execute robot grasps on transparent objects.
While NeRF was originally proposed as an alternative for explicit volumetric representations and shown to render novel views of complex scenes realistically [1], it can also reconstruct the scene geometry.
In particular, due to the view-dependent nature of the NeRF model, it can learn to represent the geometry associated with transparency accurately.
The only input requirement to train a NeRF model is a set of images taken from a camera with known intrinsics (e.g., focal length, distortion) and extrinsics (position and orientation in the world).
While the intrinsics can be determined from calibration techniques or from the camera itself, determining the extrinsics is often a challenge [2, 3]. However, robots operating in a ﬁxed workcell or with a camera mounted on the manipulator arm, can readily determine camera intrinsics.
This makes NeRF a particularly good match for robot manipulators.
DatasetGeneration
[Figure]
In experiments, we show qualitatively and quantitatively that Dex-NeRF can compute high accuracy depth images from photo-realistic synthetic and real scenes, and achieve 90 % or better grasp-success rates on real objects.
The contributions of this paper are: (1) integration of NeRF with robot grasp planning, (2) a transparency-aware depth rendering method for NeRF, (3) experiments on synthetic and real images showing NeRF with Dex-Net generates high-quality grasps, (4) synthetic and real image datasets with transparent objects for training NeRF models.
Related Work
Detecting Transparent Objects For robots to interact with transparent objects, they must ﬁrst be able to detect them.
The most recent approaches detecting and recognizing transparent objects are data-driven.
Lai et al.
[4] and Khaing et al.
[5] propose using a Convolutional Neural Network (CNN) to detect transparent objects in RGB images.
Recently, Xie et al.
[6] developed a transformerbased pipeline [7] for transparent object segmentation.
Other methods rely on deep-learning models to predict the object pose.
Phillips et al.
[8] trained a random forest to detect the contours of transparent objects for pose estimation and shape recovery.
Xu et al.
[9] proposed a two-stage method for estimating the 6-degrees-of-freedom (DOF) pose of a transparent object from a single RGBD image by replacing the noisy depth values with estimated values and training a DenseFusion-like network structure [10].
Sajjan et al.
[11] extend this and incorporate a neural network trained for 3D pose estimation of transparent objects in a robotic picking pipeline. Zhou et al.
[12, 13] train a grasp planner directly on raw images from a light-ﬁeld camera.
Zhu et al.
[14] used an implicit function to complete missing depth given noisy RGBD observation of transparent objects.
However, these data-driven methods rely on large annotated datasets that are hard to curate, whereas Dex-NeRF does not require any prior dataset.
Neural Radiance Fields Recently, implicit neural representations have led to signiﬁcant progress in 3D object shape representation [15, 16, 17] and encoding the geometry and appearance of 3D scenes [18, 1].
Mildenhall et al.
[1] presented Neural Radiance Fields (NeRF), a neural network whose input is a 3D coordinate with an associated view direction, and output is the volume density and view-dependent emitted radiance.
Due to its view-dependent prediction, NeRF can represent non-Lambertian effects such as specularities and reﬂections, and therefore capture the geometry of transparent objects.
However, NeRF is slow to train and has low data efﬁciency.
Yu et al.
[19] proposed Plenoctrees , mapping coordinates to spherical harmonic coefﬁcients, shifting the viewdependency from the input to the output.
In addition, Plenoctrees pre-samples the model into a sparse octree structure, achieving a signiﬁcant speedup in training over NeRF.
Deng et al.
[20] proposed JaxNeRF, an efﬁcient JAX implementation of NeRF reduces the training time of a NeRF model from over a day to several hours.
Deng et al.
[21] add depth supervision to train NeRF 2 to 6 × faster given fewer training views.
Adamkiewicz et al. [22] proposed an algorithm that uses a NeRF model for robot navigation. In this work, we propose to use NeRF to recover the geometry of transparent objects for the purpose of robotic manipulation.
Robotic Grasping Traditional robot grasping methods analyze the object shape to identify successful grasp poses [23, 24, 25].
Data-driven approaches learn a prior using labeled data [26, 27] or through self-supervision over many trials in a simulated or physical environment [28, 29] and generalize to grasping novel objects with unknown geometry.
These approaches rely on RGB and depth sensors to generate an accurate observation of the target object. Additionally, different methods use different inputs, such as depth maps [30, 31, 32], point clouds [33, 34, 35, 9], octrees [36], or a truncated signed distance function (TSDF) [37, 38].
In contrast, in this paper we propose a method to render a high-quality depth map from a NeRF model to then pass to Dex-Net [30] to compute a grasp.
While standard depth cameras have gaps in their depth information that needs to be processed out with hole-ﬁlling techniques, the depth map rendering from NeRF is directly usable.
It is possible that other grasp-planning techniques may be able to plan grasps from NeRF models.
Problem Statement
We assume an environment that has an array of cameras at ﬁxed known locations, or that the robot can manipulate a camera (e.g., wrist-mounted) to capture multiple images of the scene.
Given the environment contains rigid transparent objects, compute a frame for a robot gripper that will result in a stable grasp of a transparent object.
Method
In this section, we provide a brief background on NeRF, then describe recovering geometry of transparent objects, integrating with grasp analysis, and improving performance with additional lights.
4.1 Preliminary: Training NeRF
NeRF [1] learns a neural scene representation that maps a 5D coordinate containing a spatial location ( x, y, z ) and viewing direction ( θ, φ ) to the volume density σ and RGB color c .
Training
NeRF’s multilayer perceptron (MLP) requires multi-view RGB images of a static scene with their corresponding camera poses and intrinsic parameters.
The expected color C ( r ) of the camera ray r = o + t d between near and far scene bounds t n and t f is:
[Equation]
where T ( t ) = exp (cid:16) − (cid:82) t t n σ ( r ( s ))
ds (cid:17) is the probability that the camera ray travels from near bound t n to point t without hitting any surface.
NeRF approximates the expected color ˆ C ( r ) as:
[Equation]
where T i = exp (cid:16) − (cid:80) i − 1 j =1 σ j δ j (cid:17) and δ i
=
t i +1 − t
i is the distance between consecutive samples on the ray r .
During training, NeRF minimizes the error between the rendered and ground truth rays’ colors using gradient descent.
4.2 Recovering Geometry of Transparent Objects
We observe that NeRF does not directly support transparent object effects—it casts a single ray per source image pixel without reﬂection, splitting, or bouncing.
The incorporation of the viewingdirection in its regression and supervising with view-dependent emitted radiance allows recovery of non-Lambertian effects such as reﬂections from a specular surface.
However, while RGB color c is view-dependent, the volume density σ is not—meaning NeRF has to learn a non-zero σ to represent any color at that spatial location.
Visually, the usual result is that the transparent object shows up as a “ghostly” or “blurry” version of the original object. [Figure] [Figure]
When training [Figure] a NeRF model learns a density σ of each spatial location.
This density corresponds to the transparency of the point
[Figure]
and serves to help learn how much a spatial location contributes to the color of a ray cast through it.
Although NeRF converts each σ i to an occupancy probability α i = 1 − exp( − σ i δ i ) , where δ i is the distance between integration times along the ray, thus implicitly giving α i an upper bound of 1, it does not place a bound on the raw σ value.
We use the raw value of σ to determine if a point in space is occupied.
4.3 Rendering Depth for Grasp Analysis
We propose using Dex-Net to compute grasps on transparent objects.
Dex-Net computes candidate grasp poses given a depth image of a scene.
To generate a depth image, we consider two candidate reconstructions of depth.
First, we use the same rendering of sampled points along a camera ray that
NeRF uses.
This Vanilla NeRF reconstruction ﬁrst converts σ i to an occupancy probability α i .
It then applies the transformation w i =
α i (cid:81) i − 1 j =1 (1 − α j ) .
To render depth at pixel coordinate [ u, v ] , it computes the sum of sample distances from the camera weighted by the termination probability
D [ u, v ] =
(cid:80) Ni
=1 w i
δ i .
When applied on transparent objects, however, this results in noisy depth maps, as shown in Fig.
Instead, we consider a second, transparency-aware method that searches for the ﬁrst sample along the ray for which σ i > m where m is a ﬁxed threshold.
The depth is then set to the distance of that sample δ i .
We explore different values for m , and observe that low values result in a noisy depth map while high values create holes in the depth map.
In our experiments we set m = 15 (see Fig.
4.4 Improving Reconstruction with Light Placement
For NeRF to learn the geometry of a transparent object, it must be able to “see” it from multiple camera views.
If the transparent object is not visible from any views, then it will have no effect on the loss function used in training, and thus not be learned.
We thus look for a way to improve visibility of transparent objects to NeRF.
One property that transparent objects share (e.g., glass, clear plastic) is that they are glossy and thus produce specular reﬂections when the camera view direction is opposite to the surface normal of the incident direction of light.
To the NeRF model, a specular reﬂection viewed from multiple angles will appear as a white point on a solid surface—i.e., c
=
[1 , 1 , 1] T and σ > 0 , while from other angles it will appear as σ ≤ 0 .
As σ is view-independent, NeRF learns a σ between fully opaque and fully transparent for such points.
By placing additional lights in the scene, we create more angles from which cameras will see specular reﬂections from transparent objects—this results in NeRF learning a model that ﬁlls holes in the scene.
While the number and placement of lights for optimal training is dependent on both the expected object distribution and camera placement, in experiments (Sec. 5.5) we show that increasing from 1 light to a 5x5 array of lights improves the quality of the learned geometry.
Experiments
We experiment in both simulation and on a physical ABB YuMi robot.
We generate multiple datasets, where each dataset consists of images and associated camera transforms of one static scene containing one or more transparent objects.
We train NeRF models using a modiﬁed JaxNeRF [20] implementation on 4 Nvidia V100 GPUs.
We use an existing pre-trained Dex-Net model for grasp planning without modiﬁcation or ﬁne-tuning.
We can do this since NeRF models can be rendered to depth maps from arbitrary camera intrinsics and extrinsics, thus we match our NeRF rendering to the Dex-Net model instead of training a new one.
5.1 Datasets
As existing NeRF datasets do not include transparent objects, and existing transparent-objectgrasping datasets do not include multiple camera angles, we generate new datasets using 3 different methods: synthetic, Cannon EOS 60D camera with a Tamron Di II lens with a locked focal length, and an Intel RealSense.
For synthetic datasets, we use Blender 2.92’s physically-based Cycles renderer with path tracing set to 10240 samples per pixel, and max light path bounces set to 1024.
We chose theses settings by increasing them until renderings were indistinguishable from the previous setting—ﬁnding that lower settings lead to dark regions and smaller specular reﬂections.
For glass materials, we set the index of refraction to 1.45 to match physical glass.
We include 8 synthetic datasets of transparent objects: 2 scenes with clutter: light array and single light; 4 singulated objects from Dex-Net:
Pipe Connector, Pawn, Turbine Housing, Mount; and 2 household objects: Wineglass upright and
Wineglass on side.
As these computationally demanding to render due to the high quality settings, we distribute these as part of the contribution.
For the Cannon EOS and RealSense real-world datasets, we place ArUco markers in the scene to aid in camera pose recovery and take photos around the objects using a ﬁxed ISO, f-stop, and focal length.
We use bundle adjustment from COLMAP [2, 3] to reﬁne the camera poses and intrinsics to high accuracy.
We include 8 physical datasets of transparent objects with a variety of camera poses: table with clutter, Dishwasher, Tape Dispenser, Wineglass on side, Flask, Safety Glasses, Bottle upright, Lion Figurine in clutter.
The main difﬁculty in generating these datasets is calibration and computing high-precision camera poses.
The datasets (at https://sites.google.com/view/dex-nerf ) differ from prior work in their focus on scenes with transparent objects in a graspable setting, with over 70 camera poses each.
5.2 Synthetic Grasping Experiments
We also test the ability of Dex-NeRF to generate grasps on the synthetic singulated transparent
DexNet object datasets.
For each dataset, we evaluate the grasp in simulation using a wrench resistance metric measuring the ability of the grasp to resist gravity [39].
Fig.
4 shows images of the synthetic objects, Dex-NeRF-generated depth map, and an example sampled grasp for each.
To measure the effect of training time on grasp-success rate, we simulate and record grasps over the course of training.
In Fig.
5, we observe that grasp success rate improves with training time, but plateaus
[Figure] [Figure]
[Figure] [Figure]
PhoXi
Vanilla NeRF
Dex-NeRF
Tape Dispenser 0/
Wineglass 0/ 0/ 9/
Flask 0/ 1/ 9/
Safety Glasses 0/ 0/
Bottle 0/ 10/ 10/
Lion Figurine 0/ 3/ 10/ between 80 % and 98 % success rate at around 50k to 60k iterations. This suggests that there may be a practical ﬁxed iteration limit to obtain high grasp success rates.
We test Dex-NeRF on a scene of a tabletop cluttered with transparent objects.
In this experiment, the goal is to grasp a transparent object placed in a stable pose in close proximity to other transparent objects.
The challenge is twofold: the depth rendering quality should be sufﬁcient for both grasp planning and collision avoidance.
Fig.
1 shows the robot and scene in the upper left, and the overhead image, depth, and computed grasp inline in the pipeline, and the ﬁnal computed grasp with simulated execution is in the upper middle image.
The ﬁnal grasp contact point was accurate to a 2 mm tolerance, suggesting that Dex-NeRF with sufﬁcient images taken from precisely-known camera locations may be practical in highly cluttered environments.
5.3 Physical Grasping Experiments
To test the Dex-NeRF in a physical setup, we place transparent singulated objects in front of an
ABB YuMi robot, and have the robot perform the computed grasps.
We compare to 2 baselines: (1) PhoXi , in which a PhoXi camera provides the depth map; and (2) Vanilla NeRF , in which we use the original depth rendering from NeRF.
The PhoXi camera is normally able to generate highprecision depth maps for non-transparent objects.
All methods use the same pre-trained Dex-Net model, and both Vanilla NeRF and Dex-NeRF use the same NeRF model—the only difference is the depth rendering.
We test with 6 objects (Fig. 6), and compute and execute 10 different grasps for each and record the success rate.
A grasp is successful if the robot lifts the object.
In Table 1, we see that Dex-NeRF gets 90 % and 100 % success rates for all objects, while the baselines get few successful grasps.
The PhoXi camera is unable to recover any meaningful geometry which causes Dex-Net predictions to fail.
The Vanilla NeRF depth maps often have unpredictable protrusions that result in Dex-Net generating unreliable grasps. [Figure] [Equation]
5.4 Comparison to RealSense Depth
We qualitatively compare the rendered depth map of the proposed pipeline against a readily-available depth sensor on scenes with transparent objects in real-world settings (Fig. 2).
We select the Intel
RealSense as it is common to robotics applications, readily available and high-performance.
The
RealSense, like most stereo depth cameras, struggles with transparent objects as they are unable to compute a stereo disparity between pixels from different cameras when the pixels are specular reﬂections or the color of the object behind the transparent object.
The RealSense optionally projects a structured light pattern on the scene to aid in computing depth from textureless surfaces; however, in experiments, we observed no qualitative difference with and without the light pattern emitter enabled.
We use a Canon EOS for NeRF, and use a RealSense for a depth image.
In this experiment, we observe that the RealSense cannot compute the depth of most transparent objects and often produces regions of unknown depth (shown in black) where transparent objects are.
On the other hand, the proposed pipeline produces high-quality depth maps with only a few noisy areas.
5.5 One vs Many Lights
We experiment with different light setups to test the effect of specular reﬂections on the ability of
NeRF to recover the geometry of transparent objects.
We create two scenes (Fig. 7), one with a single bright light source directly above the work surface, and another with an array of 5x5 (25) lights above the work surface.
We set the total wattage of the lights in each scene to be the same.
Since most lights in the multiple light scene are further away from the work surface than the single light source, the scene appears darker, though more evenly illuminated.
The effect of the specular reﬂections is prominent on the lightbulb in the lower part of the image.
In the single light source, there is a single specular reﬂection, while in the multiple light scene, the reﬂection of the array of lights is visible.
With the same camera setup for both scenes, we train NeRF models with the same number of iterations.
We show the depth rendering in Fig.
7 and circle a glass and a wineglass on their side.
In the single-light source image, the closer surfaces of the glasses are missing, while in the multiple-light source depth image, the glasses are nearly fully recovered.
This suggests that additional lights in the scene can help NeRF recover the geometry of transparent objects better.
[Figure]
5.6 Workcell Setup
We experiment with a potential setup for a robot workcell in which a grid of overhead cameras captures views of the cluttered scene so that a robot manipulator arm can then perform tasks with transparent objects in the workcell.
We propose that a grid of overhead cameras would be practical to setup and would not obstruct manipulator tasks nor operator interventions.
The objective is to determine how many overhead cameras would be needed to recover a depth map of sufﬁcient accuracy to perform manipulation tasks.
We place a 2 m by 2 m grid of cameras 1 m above the work surface, and have them all point at the center of the work surface.
Each camera has the same intrinsics, and are evenly spaced along the grid.
We experiment with grids having 4, 9, 16, 25, 36, and 49 cameras.
The environment has the same 5x5 grid of lights as before.
For each camera grid, we train JaxNeRF for 50k iterations and compare performance.
After training, we observe increasing peak signal to noise ratios (PSNR) and structural similarity (SSIM) scores with increasing number of cameras.
The 2x2 grid of cameras produces a high train-totest ratio for PSNR, likely indicating overﬁtting to training data, and results in a depth map without apparent geometry.
This ratio decreases with additional cameras.
The minimum number of cameras for this proposed setup appears to be around 9 (3x3) as its depth map is usable for grasp planning, while the 5x5 grid shows better PSNR and SSIM and ratio between train and test PSNR, and the 7x grid is the best.
See Fig.
9 for a visual comparison.
Additionally, we trained 9x9, 11x11, and 13x grids, observing no statistically signiﬁcant improvement beyond the 7x7 grid.
Conclusion
In this work, we showed that NeRF can recover the geometry of transparent objects with sufﬁcient accuracy to compute grasps for robot manipulation tasks.
NeRF learns the density of all points in space, which corresponds to how much the view-dependent color of each point contributes to rays passing through it.
With the key observation that specular reﬂections on transparent objects cause NeRF to learn a non-zero density, we recover the geometry of transparent objects through a combination of additional lights to create specular reﬂections and thresholding to ﬁnd transparent points that are visible from some view directions.
With the geometry recovered, we pass it to a grasp planner, and show that the recovered geometry is sufﬁcient to compute a grasp, and accurate enough to achieve 90 % and 100 % grasp success rates in physical experiments on an ABB YuMi robot.
We created synthetic and real datasets for experiments in transparent geometry recovery, but we believe these datasets may be of interest to researchers interested in extending NeRF capabilities in other ways and thus contribute them as part of this research project.
Finally, to test if NeRF could be used in a robot workcell, we experimented with grids of cameras facing a worksurface and their ability to recover geometry in potential setup, and showed the increased capabilities and point of diminishing return for additional cameras.
In future work, we hope to address one of the main drawbacks of NeRF—the long training time required to obtain a NeRF model.
Many research groups have sped up training time through improved implementations, new algorithms, new network architectures, pre-conditioned network weights, focused sampling, and more.
While these approaches apply to general NeRF training, we plan to exploit features speciﬁc to robot scenerios to speed up training, including using depth camera data as additional training data, manipulator-arm-mounted cameras to inspect regions of interest, and visio-spatial foresight to adapt to changes in the environment. Acknowledgments
This research was performed at the AUTOLAB at UC Berkeley in afﬁliation with the Berkeley AI
Research (BAIR) Lab, Berkeley Deep Drive (BDD), the Real-Time Intelligent Secure Execution (RISE) Lab and the CITRIS “People and Robots” (CPAR) Initiative.
We thank our colleagues who provided helpful feedback and suggestions, in particular Matthew Tancik.
This article solely reﬂects the opinions and conclusions of its authors and do not reﬂect the views of the sponsors or their associated entities.