Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects
Anonymous Author(s)
AfﬁliationAddress email
Abstract: The basis of many object manipulation algorithms is RGB-D input.
Yet,commodity RGB-D sensors can only provide distorted depth maps for a wide rangeof transparent objects due light refraction and absorption.
To tackle the perceptionchallenges posed by transparent objects, we propose TranspareNet, a joint pointcloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially ﬁlled ﬂuid contents within the vessels.
To address the shortcomings of existing transparent objectdata collection schemes in literature, we also propose an automated dataset creationworkﬂow that consists of robot-controlled image collection and vision-based automatic annotation.
Through this automated workﬂow, we created Transparent Object
Depth Dataset (TODD), which consists of nearly 15000 RGB-D images.
Our experimental evaluation demonstrates that TranspareNet outperforms existing state-of-theart depth completion methods on multiple datasets, including ClearGrasp, and thatit also handles cluttered scenes when trained on TODD.
Code and dataset will bereleased at https://anonymous.4open.science/r/TranspareNet-567C/
Introduction
RGB-D sensors have been instrumental in 3D perception for robotics.
While reliable for opaqueobjects, commodity-level RGB-D sensors often fail when capturing the depth of transparent objects,made of materials such as glass or clear plastic.
This is because transparent objects possess uniquevisual properties that distort the captured depth.
Specular surface reﬂections introduce gaps in thedepth map, while depth projection to surfaces behind the transparent object, as shown in Figure 1,introduce inaccurate depth estimates.
Nonetheless, transparent objects are common in our homesand daily lives, from kitchens and dining rooms to laboratory settings.
Perception of these objects isunder-explored, yet vital for robotic systems that operate in unstructured human-made environments.
In this work, we present a method to leverage real complex transparent object depth to estimateaccurate 3D geometries of transparent objects.
The design is motivated by three main ideas.
First,although the depth information for transparent objects captured by RGB-D sensors is noisy by nature,there remains useful components that can be leveraged by downstream depth-completion tasks.
Theprevious state-of-the-art method and dataset, ClearGrasp [1], masks out all depths at the location ofthe transparent object, renders them invalid, and conducts global optimization with surface normaland boundaries to reconstruct its geometry.
This is sub-optimal, as it ignores all depth informationat the transparent object’s location.
Ours leverages the unique depth distortion at the location oftransparent objects to generate a point cloud distribution using the Point Cloud Completion module,which is then fed through a Depth Completion module to generate the complete depth map.
Second,existing large-scale datasets for transparent objects are either synthetic [1], too simple and withoutclutter [2], or lack depth information [3].
There remains a lack of datasets for transparent objectsin a real-world setting with clutter and content within.
Therefore, we introduce a novel Transparent
Object Depth Dataset (TODD) which is comparable in scale to existing transparent object datasets,and captures the realistic state that transparent objects are found in everyday life, including clutter,which is absent from previous datasets.
Finally, collecting a dataset of transparent objects has eitherinvolved synthetic generation [1], which has a signiﬁcant and often insurmountable domain gap with real world data, or manual placement and capture, which is too difﬁcult to scale up in size.
Therefore,we design an automated pipeline for dataset collection, which can capture and annotate RGB-D data,including the corrected depth for transparent objects.
Our primary contributions are threefold:
We introduce the Transparent Object Depth Dataset (TODD), a large-scale real transparentobject dataset with around 15,000 RGB-D observations that contain RGB, raw depthspaired with ground truth depth (in which the depth at the transparent object is complete),instance segmentation, and object pose.
To our knowledge, our dataset is the ﬁrst large-scale
transparent object dataset that contains complex scenes with clutter and contains transparent
objects in a realistic setting with ﬂuid content within the vessels.
We introduce a scalable automatic dataset collection and annotation scheme for the collection
of RGB-D images of transparent objects, with fully automatic labeling of ground truth depth,
6DoF pose, as well as instance segmentation.
We introduce a novel depth completion method, TranspareNet, that achieves state-of-the-art
performance on the existing ClearGrasp dataset [1] and benchmark its performance on
our dataset, TODD.
TranspareNet is a joint point cloud and depth completion method thatleverages RGB and depth signals of transparent objects.
Related Work
Transparent Object Segmentation.
Transparent objects’ refractive and reﬂective nature, as well aschanging appearance in various scenes due to background, makes detecting them a challenging taskin computer vision.
There have been several works that applies state-of-the-art segmentation methods[4, 5, 6] to transparent object segmentation [7, 1, 8].
However, due to the signiﬁcant domain gapbetween opaque and transparent objects, all of these aforementioned methods were trained on customtransparent object datasets.
Due to the cost associated with image collection and annotation, manysynthetic datasets have been proposed, including TOM-Net [9], ClearGrasp [1], and Omni [10] 1 .
While real world transparent object datasets, like Trans10K [8] and LabPic [7], avoid the potentialdomain gap between rendered and real images, they are time-consuming and expensive to collect andannotate.
LIT [11] utilizes a special light-ﬁeld sensor to detect transparent objects and estimate theirposes.
Our proposed automated dataset collection pipeline enables large scale, automatic annotationof transparent objects using common RGB-D sensor.
Depth Completion.
Depth completion refers to the task of generating a dense depth map from anincomplete depth map due to sparse measurements, noise, or sensor limitations.
In the context of ourwork, we use depth completion to describe the task of completing and reﬁning noisy depths capturedby RGB-D sensors for transparent objects.
Works in this area generally fall in four main schemes:constraints driven, monocular depth estimation, depth completion from sparse point cloud, and depthcompletion given noisy RGB
-D. Our work falls in the last scheme.
Constraint driven approachesassume a speciﬁc setup method with ﬁxed viewpoint(s) and capturing procedure [12, 13, 14, 15, 16],sensor type [17, 18, 11], or known object models [19, 20, 21].
Our proposed depth completionmethod does not apply any assumptions.
Monocular depth estimation refers to the direct regressionof depth from RGB input [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]. This family of works
[Figure]
generally require access to large-scale RGB-D datasets, for which the depth reﬂects objects depictedin the RGB image.
Historically, this area has not been explored for transparent object handling, dueto the absence of large scale transparent data paired with noise-free depth.
Our proposed dataset anddata collection and annotation approach can now facilitate the generation of sufﬁcient data to supportresearch in this direction.
The third scheme, depth completion from sparse depth data, refers to theconversion of a sparse point cloud to a dense depth map via deep learning methods.
Works in thisarea concentrate on applications involving LiDAR for outdoor environments [36, 37, 38, 39], andare inherently different from the semi-sparse and noisy signals from RGB-D sensors for transparentobjects.
There are two main lines of work in depth reﬁnement of noisy RGB-D data of transparentobjects.
ClearGrasp [1] takes an optimization approach and estimates surface normals, occlusionboundary, and segmentation to perform a global optimization for depth estimation.
More recently,[10] proposed a voxel based local implicit neural function for depth estimation.
We show that ourmethod outperforms both of these methods.
The key insight that contributed to the success of ourmethod is that we leverage the unique distortion in the depth caused by transparency to generate acoarse estimation of object depth through point cloud completion, and conduct depth completion forreﬁnement of the sparse point cloud.
Previous methods like ClearGrasp [1] discard this information.
Prior Transparent Dataset Collection and Annotation .
Trans10K[8] is a 2D transparent objectsegmentation dataset that consists of real images of transparent objects and their semantic segmentation masks.
All of its images and labels are captured and annotated manually.
Schenck and Fox[40] proposed a dataset of transparent liquid in opaque containers, and consists of rendered andreal-world images.
The captured images’ segmentation masks are annotated via thermal cameras incombination with heated liquid.
For transparent object datasets with depth information, ClearGrasp[1] uses a synthetic dataset generated by Blender as the training set, and collects the small scalereal-world validation and testing dataset using matched transparent and opaque objects.
This processrequires manual placement and matching of transparent and opaque pairs for each generated image.
Another 3D dataset, Keypose [2], collects its dataset using a eye-in-hand robot arm and trackingsystem based on AprilTag 2 [41].
The camera path is calculated using AprilTag, and along the scantrajectory, several frames’ keypoints are manually labelled, where the rest are labeled based on thetrajectory.
The depth info is collected using a similar replacing method as ClearGrasp [1].
Suchreplacing method needs human intervention to align the opaque and transparent twins through imageoverlay, which can be hard and inaccurate especially in complex scenes.
Furthermore, since eachimage in ClearGrasp [1] is manually captured, opaque swapped and annotated, it cannot scale to alarger dataset size due to time and labour constraints.
Our method overcomes the manual effort that previous methods used for dataset collection with an automated pipeline for dataset creation and annotation.
Transparent Object Depth Dataset (TODD)
[Figure] We use an eye-in-hand Franka Emika Panda controlled by FrankaPy[42] with Intel Realsense RGB-Dcamera to collect the dataset.
As shown in Figure 3, the pipeline consists of multiple steps.
First,the end-effector of the robot arm moves the camera to multiple positions around the transparentobject(s), while ensuring focus on the transparent object.
Templates with AprilTags [41] andalignment marks are used to maintain the known translation between objects and tags.
Then for everyviewpoint captured, AprilTags can be recognized together with their 6DoF pose in camera coordinates.
Combining this with the known shift between tags and transparent objects, the corresponding CADmodel of each object can be overlaid at the appropriate locations of transparent objects within theimage.
The resultant 3D meshes are sufﬁcient to automate the succeeding dataset annotation tasks.
Namely, projection of meshes to the image space provides the instance segmentation mask as wellas the ground truth depth as shown in Figure 4.
With our proposed dataset creation pipeline, wecan collect and annotate 300 RGB-D images for one scene in 30 minutes with minimal humanintervention.
Compared to methods used by [1, 3, 2], our proposed pipeline only needs placingthe transparent object once per sequence and annotation is fully automated, which guarantees theaccuracy of annotated labels.
In total, TODD has 14,659 images of scenes which contains six glass beakers and ﬂasks in ﬁvedifferent backgrounds.
Four objects are used in training set and the other two novel objects form
ClearGrasp[1]
Trans10K[3]
KeyPose[2]
Ours
Real Samples 0.3 × 1010 × 1040 × 1015 × 10 3 Auto Collection & Annotation (cid:56) (cid:56) (cid:56) (cid:52)
[Figure]
[Table]
Depth (cid:52) (cid:56) (cid:52) (cid:52)
Instance
[Table]
Pose
[Table]
the novel validation and test set.
The training set has 10,302 images where validation and testingset combined has 4357 images.
When comparing with existing datasets in Figure 5, TODD has thefollowing advantages.
Every scene consists of up to three transparent objects with occlusion, whichintroduces additional complexity to the dataset compared to KeyPose’s [2] single object scenes.
Theobjects and their placement are selected to mimic real-life transparent glassware, which can help todevelop vision aware robots capable of manipulating transparent vessels.
One particular directionof that is robotic chemists automating chemical research [43].
Additionally, the glass vessels in thedataset are empty or ﬁlled with 5 different coloured liquids to simulate real-life circumstances, whichis common in 2D transparent datasets like Trans10K [8] and LabPic [7], yet is missing in 3D datasetslike ClearGrasp [1] and KeyPose [2].
Besides the RGB and raw depth information, the datasetconsists of ground truth depth, instance segmentation mask, as well as the objects’ 6DoF poses.
Ourdataset’s size is much bigger than ClearGrasp’s [1] 286 real world images, and is comparable with
Trans10K (10K) [8] and KeyPose(48K)[2], while having the potential to quickly scale to greater sizewith relative ease.
Our Depth Completion Method
Existing transparent object depth completion and pose estimation methods consider the distortedand incomplete depth of transparent objects as misguiding information and either only rely on colorinformation [2], or crop out the incorrect depth [1].
We propose a novel depth completion method tohandle transparent objects called TranspareNet, whose general pipeline is shown in Figure 2.
Insteadof discarding the transparent objects’ depth, each item’s depth is separated out and de-projected intoa point cloud.
With the distorted point cloud, a Point Cloud Completion network is used to recoverthe basic shape in the form of a predicted point cloud.
However, such a point cloud is still too sparseand noisy to be considered as the ﬁnal complete depth and used in downstream manipulation tasks.
To further reﬁne the depth, predicted point clouds are projected back to the depth channel and a depthcompletion module is used to ﬁll the blank depth as well as correct the shifted depth.
By combiningboth point cloud and depth completion tasks, our method avoids directly feeding incorrect transparentdepth to the depth completion task, while capable to utilize the unique distortion depth caused bytransparency and generate a coarse estimation of object depth via point cloud completion.
Suchintermediate depth improves the performance of the downstream depth completion subtask, as shownin Section 5.
4.1 Point Cloud Completion
Transparent objects cause the RGB-D sensor to report invalid or inaccurate depth information.
The
Point Cloud Completion module is aimed at taking each object’s depth as point cloud and estimatingthe correct depth by predicting the completed point cloud.
Inspired by the point cloud processingmodules proposed in GRNet [44], we ﬁrst feed the incomplete point cloud from transparent objectdepth de-projection through a Gridding layer [44], which computes weighted vertices of 3D gridcells that points lie in.
Then we use a 3D CNN encoder-decoder with U-Net connections [45] tolearn the features that are necessary for point cloud completion.
The succeeding component is the
Gridding reverse layer [44], which back projects each 3D grid cell to a point in the coarse complete [Table] cloud whose coordinate is the weighted sum of the eight vertices.
This is then forward propagatedthrough a Multi-Layer Perceptron that creates the ﬁnal completed point cloud.
For each point, itscorresponding grid cell’s features from the CNN decoder are concatenated and combined with it.
The
MLP also reﬁnes the coarse to ﬁnal point cloud with the aim of recovering details of the target object.
However, the 3D CNN based network’s limited resolution leads to sparse and noisy prediction, thusan additional depth completion module is needed.
4.2 Depth Completion
The output from the depth completion module consists of a sparse distribution around the location ofthe transparent vessel(s) shown in Figure 2.
We fuse this with the RGB signal to generate a 4D inputto our encoder-decoder structure.
The encoder-decoder based depth-completion module processes thescene’s depth to ﬁll in the sparse depth and correct the noisy depth present.
The decoder of the Depth
Completion module consists of spatially-adaptive denormalization (SPADE) blocks, ﬁrst introducedin [46].
Our usage of SPADE in the encoder-decoder Depth Completion module is a variant of [47].
This module enables us to learn spatially-dependent scale and bias for decoder feature maps, whichhelps reduce the domain shift between RGB and depth, as introduced by the empty depths on thedepth map.
Let m denote the object mask.
Given a batch of N inputs with dimension C i × H i × W i ,the output of the SPADE block at site ( n ∈ N , c ∈ C
i , y ∈ H i , x ∈ W i ) is then
[Equation]
) h in,c,y,x is the activation at the site prior to normalization, µ ic and σ ic are the mean and standarddeviation of the activations within channel c .
γ ic,y,x ( m ) and β ic,y,x ( m ) represent the modulationparameters.
They represent the scaling and bias values for the i -th activation map at site ( c, y, x ) ,respectively.
4.3 Loss Function
The point cloud completion network is trained with Gridding Loss [44], which is a L1 distancebetween predicted G p = <
V p , W p > and ground truth G gt = <
V gt , W gt > 3D grids in N Gresolution.
V =
{ v i } N 3 G i =1 is collection of all vertices in 3D grid and W = { w i } N 3 G i
=1 is the weightscorresponding to each vertex.
Gridding Loss bypasses the un-orderedness of point clouds and isevaluated on the 3D grid.
The depth completion network is trained using log L 1 pair-wise loss whichforces the pairs of pixels in the predicted depth to regress to similar values as the correspondingpairs in the ground truth depth [47].
Let G describe the set of pixels where the ground truth depthis non-zero, i and j are the pixel pairs, and y and y ∗ denote the ground truth and predicted depths,respectively.
We express these two loss functions as:
[Equation]
Experimental Results
We conduct experiments to test our TranspareNet model along with SOTA methods like NLSPN[48] and ClearGrasp[1] as well as superior con-current method LocalImplicit [10] on the ClearGrasp [1] dataset to demonstrate our method’s strong performance.
Additionally, we test ClearGrasp,
TranspareNet, as well as its point cloud and depth completion modules on our TODD dataset.
Thisallows us to study the effects of each individual module of TranspareNet and advantages of theproposed joint pipeline over existing methods for depth and point cloud completion.
Importantly, ourresults demonstrate that distorted transparent object depth can be converted into sparse depth estimatevia point cloud completion and used to improve downstream depth completion quality.
5.1 Metrics
For depth completion, the standard metrics as described in [1] are followed.
The prediction andground truth arrays are ﬁrst resized to 144 × 256 resolution prior to evaluation.
Errors are computedusing the following metrics, Root Mean Squared Error (RMSE), Absolute Relative Difference (REL),
Mean Absolute Error (MAE) and Threshold ( δ ).
[Equation]
Threshold is % of d i ∈
ˆ D satisfying max( d i d ∗ i , d ∗ i d i ) <
δ .
Here, ˆ D denotes the set of pixels G ∩ D , whichhas a valid corresponding ground-truth depth, and falls within the mask.
d i ∈
ˆ D is the predicteddepth, and d ∗ i is the corresponding ground-truth depth.
Note that RMSE, REL and MAE metrics arecomputed in meters.
For the threshold metric, δ is set to 1.05, 1.10 and 1.25.
5.2 Depth Completion Comparison with State-of-the-Art Methods
The ability of our method to estimate depth for transparent objects is shown Table 2.
All modelswere trained using the ClearGrasp [1] dataset.
Seen denotes synthetic objects which have appearedin training set, under a different setting.
Unseen denotes synthetic objects that have not appeared inthe training set.
We compare with several state-of-the-art methods.
NLSPN [48] is a state-of-the-artmethod for depth completion on the NYUV2 [49] and KITTI [37] datasets.
ClearGrasp [1] is thestate-of-the art of transparent object depth completion using global optimization.
LocalImplicit[10] is a new con-current method that achieved superior performance than ClearGrasp under severalmetrics.
The Point Cloud Completion (PCC) module cannot be trained using the ClearGrasp syntheticdataset, since it lacks raw sensor depth.
Therefore, we only show the performance of the Depth
Completion (DC) component of our proposed method and demonstrate that it already outperforms alldepth completion methods for the metrics presented, by a large margin.
5.3 Performance on Our Dataset
We evaluate each component of our model on our proposed dataset as well as compare the resultswith performance of the ClearGrasp algorithm [1].
Evaluation of ClearGrasp uses the pre-pretrainedmodel, since we cannot re-train ClearGrasp [1] on our data, as ground truth surface normal and
[Table]
occlusion boundaries are non-trivial to obtain for our real-object dataset TODD.
For evaluation ofthe pretrained ClearGrasp model, we use the ground truth depth and mask 2 .
When we evaluatethe direct output from our Point Cloud Completion (PCC) module, we see that it suffers due to thesparsity of depths in the transparent object region, empty depth values of which are interpreted aszero to ensure numerical stability.
When computing metrics using only regions within the objectmask where the depth is non-empty, we see improved metric values (See Appendix), this meansthat the Point Cloud Completion produces meaningful depths that is valuable for the downstreamdepth completion task.
We cannot directly use the output from the Point Cloud Completion module,since the point cloud is too sparse.
When only the depth completion (DC) module is used withoutpoint cloud completion, we see inferior results as compared to our joint point cloud and depthcompletion network, TranspareNet, shown in Table 3.
We can see that TranspareNet outperforms
ClearGrasp [1], PCC, and DC.
This means that our joint approach, which involves leveraging theunique depth distortion around transparent objects, generating a course estimation, and conductingdepth completion using the point cloud distribution is indeed effective.
A qualitative comparison ofdifferent stages of our model is shown in Figure 6.
We see that for single and multi-object clutteredscenes, our method achieves SOTA results.
Conclusion
We introduced TranspareNet, a novel method that achieves state-of-the-art performance on synthetictransparent object data from ClearGrasp [1].
Our method leverages existing depth information atthe location of transparent objects to estimate the complete depth for transparent objects.
We alsointroduced a novel real dataset, TODD, which has complex scenes with numerous vessels underocclusion.
The transparent vessels in our dataset mimics vessels found in household settings in shapeand appearance.
To our knowledge, our dataset is the ﬁrst to contain depth information of transparentvessels with partially ﬁlled liquids.
We benchmark our dataset using the proposed depth completionmethod, TranspareNet, and achieve SOTA results.
We also introduced an automatic data capture andannotation workﬂow that consists of robot-controlled image collection and vision-based automaticannotation.
We hope that this work can accelerate future research in the ﬁeld of household roboticsand handling of glassware