CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation
Abstract: Realistic simulation is key to enabling safe and scalable development of self-driving vehicles.
A core component is simulating the sensors so that the entire autonomy system can be tested in simulation.
Sensor simulation involves modeling trafﬁc participants, such as vehicles, with high quality appearance and articulated geometry, and rendering them in real time.
The self-driving industry has typically employed artists to build these assets.
However, this is expensive, slow and may not reﬂect reality.
Instead, reconstructing assets automatically from sensor data collected in the wild would provide a better path to generating a diverse and large set that provides good real-world coverage.
Nevertheless, current reconstruction approaches struggle on in-the-wild sensor data, due to its sparsity and noise.
To tackle these issues, we present CADSim , which combines part-aware object-class priors via a small set of CAD models with differentiable rendering to automatically reconstruct vehicle geometry, including articulated wheels, with highquality appearance.
Our experiments show our approach recovers more accurate shapes from sparse data compared to existing approaches.
Importantly, it also trains and renders efﬁciently.
We demonstrate our reconstructed vehicles in several applications, including accurate testing of autonomy perception systems.
(cid:53)(cid:72)(cid:68)(cid:79)(cid:16)(cid:87)(cid:76)(cid:80)(cid:72)(cid:3)(cid:85)(cid:72)(cid:81)(cid:71)(cid:72)(cid:85)(cid:76)(cid:81)(cid:74)
[Table]
(cid:22)(cid:25)(cid:19)(cid:131)(cid:3)(cid:55)(cid:72)(cid:91)(cid:87)(cid:88)(cid:85)(cid:72)(cid:71)(cid:3)(cid:57)(cid:72)(cid:75)(cid:76)(cid:70)(cid:79)(cid:72)(cid:3)(cid:36)(cid:86)(cid:86)(cid:72)(cid:87)(cid:86)
Introduction
Robots, such as self-driving vehicles (SDVs), learn to navigate environments and interact safely with other agents through experience.
It is critical for robots to handle rare or safety-critical situations which are challenging or dangerous to observe in the real world [1].
Simulation provides a solution for the robot to efﬁciently experience edge-case scenarios for evaluation in a safe and cost-effective manner.
For proper testing that covers the full space of possible scenarios, the simulator should have a diverse and large set of trafﬁc participants.
For example, vehicles have a wide variety of shapes, sizes, and appearances.
We want to ensure that the perception and autonomy systems can detect them and act appropriately in all possible situations.
Existing self-driving simulators have a limited set of objects, as each object is typically designed manually, with fully speciﬁed shape, appearance, etc.
This is a time-consuming and expensive process that does not scale.
In this work, we propose to leverage real sensor data collected by the SDV in the wild to automatically build an asset library realistic sensor simulation.
In-the-wild data is cost efﬁcient and scalable to obtain, enabling collection of a diverse set of objects.
Speciﬁcally, given a sequence of sensor data (camera images and LiDAR point clouds) collected by an SDV, our goal is to automatically create digital replicas for all nearby vehicles in the scene and directly use them in a high-ﬁdelity simulator to create many scenarios that never existed before.
To achieve high quality and scalable sensor simulation, the reconstructed assets should (i) have precise shape and photorealistic appearance; (ii) be easily editable to create new variations from the original assets, (iii) be controllable, such as the ability to manipulate the vehicle to generate new behaviors ( e.g ., wheels steering) and (iv) allow for real-time rendering, which is important for closed-loop training and large-scale evaluation [2].
Out of past works using various 3D representations for reconstruction, such as point clouds [3, 4, 5, 6], voxels [7, 8, 9], or neural implicit representations [10, 11, 12, 13], the mesh representation best achieves the desired properties [14, 15].
In this paper we utilize mesh representations since they are widely used throughout simulation [16, 17, 18] and content creation systems [19, 20], are easy to manipulate [21, 22], suitable for rigging and animation [23, 24, 25, 26], support texturing and material properties [27, 28], and are fast to render [29].
Generating meshes of nearby objects from in-the-wild sensor data is a challenging task.
The sensor data is sparse ( < 30 views) and may be noisy.
Furthermore, the objects of interest are observed from limited viewpoints and portions of the objects are unobserved, whereas we want to reconstruct the complete shape for simulation.
Previous mesh reconstruction methods tackle this task as an energy minimization problem, where given an initial sphere [30] or “mean-shape” mesh template [31, 32], the goal is to optimize the mesh to be consistent with sensor data, along with some regularization.
However, when operating on in-the-wild sensor data, these methods generate overly smoothed meshes that are inaccurate [30, 32], or have severe artifacts and self-intersecting edges [33].
We demonstrate in our experiments that this is due to poor initialization.
Moreover, these approaches generate rigid meshes that cannot be controlled and lack articulation.
We therefore propose a simple and effective solution—to leverage CAD models of the object class to provide priors for reconstruction.
Not only do CAD models have high quality shape, they encode rich semantic information, such as where the wheels in a vehicle are and how they move.
This enables better initialization and reconstruction of high-quality articulated meshes from real-world data.
CAD models alone are not sufﬁcient for realistic sensor simulation, as we we want to create digital twins of any possible vehicle.
Instead, we optimize our vehicle CAD model representation with a state-of-the-art differentiable renderer in an energy minimization framework.
This allows us to faithfully reconstruct a diverse set of real-world vehicles, including their animatable parts ( e.g ., wheels), from very sparse and noisy observations.
We compare against a wide range of reconstruction methods, including recent neural rendering-based approaches, and show improved reconstruction performance on both camera and LiDAR sensor data.
We show that our reconstructed vehicles can be used for a variety of applications such as novel view synthesis, texture transfer, mixed reality via injecting simulated actors into real data, and vehicle re-animation (Fig. 1).
We also demonstrate that our approach creates a more realistic simulation environment for robots such as SDVs by evaluating perception models on simulated sensor data and showing small domain gap compared to real data.
Related Work
3D Representations for Sensor Simulation.
Triangular meshes are currently the de facto standard in real-time graphics [14, 15, 16].
While fast, ﬂexible, compact, and widely adopted in game engines and simulators [16], mesh-based assets are usually regarded as expensive to create due to the need for human artists.
Surfels [34] have also been leveraged in LiDAR simulation for robotic applications [35, 36].
However, their lack of geometric structure makes them difﬁcult to animate, edit, and compress.
Recently, volumetric representations have experienced a surge in popularity [37, 38, 39] thanks to their simplicity, differentiability, and ﬂexibility in modeling nonLambertian effects [40, 41, 42].
These works can roughly be divided into neural volumetric rendering, such as NeRFs [40, 43, 44, 45, 46], and implicit surfaces such as signed distance ﬁelds [47, 48, 49] or occupancy ﬁelds [10, 50, 51].
Various methods have been proposed to speed up surface and volumetric rendering [52, 53, 54, 55, 56].
However, these approaches still lag behind traditional rasterization pipeline in terms of rendering speed and throughput.
In our work, we use triangular meshes optimized from CAD models for efﬁcient and accurate geometry modelling.
Multi-View 3D Reconstruction in the Wild.
Despite the incredible success achieved by recent neural volumetric representations, these works are mainly demonstrated on synthetic datasets with
[Table]
(cid:39)(cid:76)(cid:73)(cid:73)(cid:72)(cid:85)(cid:72)(cid:81)(cid:87)(cid:76)(cid:68)(cid:69)(cid:79)(cid:72)(cid:3)(cid:53)(cid:72)(cid:81)(cid:71)(cid:72)(cid:85)(cid:72)(cid:85) (cid:39)(cid:68)(cid:87)(cid:68)(cid:3)(cid:40)(cid:81)(cid:72)(cid:85)(cid:74)(cid:92)(cid:3) (cid:47)(cid:68)(cid:87)(cid:72)(cid:81)(cid:87)(cid:3)(cid:70)(cid:82)(cid:71)(cid:72) (cid:48)(cid:72)(cid:86)(cid:75)(cid:3)(cid:44)(cid:81)(cid:76)(cid:87)(cid:76)(cid:68)(cid:79)(cid:76)(cid:93)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81) hundreds of (noise-free) images and dense coverage on the camera viewpoints [40, 57, 30].
However, the input is usually much sparser ( e.g ., few views of single object with similar view points) and noisier ( e.g ., localization, calibration error and LiDAR noise) for real world applications such as robotics and self-driving.
To address this issue, recent works [58, 59, 60] proposed conditional models that require expensive pre-training on other scenes with multi-view images and camera pose annotations.
Another recent line of work proposes learning mesh representations directly, either by deforming a generic mesh [61, 27, 62, 33, 63, 64], or by leveraging a differentiable mesh extraction algorithm on top of an implicit representation [65, 66, 67].
Different from previous works, we show that simple mesh optimization with differentiable rendering is sufﬁcient for high-ﬁdelity reconstruction.
CAD Priors.
Retrieving a 3D model conditioned on a single observation has been shown to be a strong baseline which outperforms more sophisticated early approaches to single-view reconstruction [68].
This motivates the use of imperfect but readily available 3D CAD models as a powerful prior for reconstruction [69, 31, 70, 71, 72, 73, 74, 75, 76].
[69, 31, 71] use the truncated SDF volumes to build the latent space for the CAD models for joint pose and shape estimation.
Uy et al.
[74, 75] encode sparse observations and use them to retrieve the most similar candidate from a CAD asset bank, which is then deformed to match the input.
While promising, none of the approaches in this area have been demonstrated to work outside of synthetic [77] datasets.
Most do not model realistic texture, nor are they part-aware.
Lu et al.
[76] conduct a two-stage CAD-based deformation and handle the wheel separately.
However, they recover the appearance only from single images without physics based material and lighting models, and do not optimize the articulation from data.
Method
Given camera images and/or LiDAR point clouds collected by a self-driving car, our goal is to automatically create digital replicas for all nearby vehicles in the scene and use them in a high-ﬁdelity simulator.
We build our model based on the observation that existing CAD models are equipped with detailed geometry and animatable parts, which can be used as a prior during the reconstruction process.
Towards this goal, we propose an energy-based formulation that exploits CAD models, as well as visual and geometric cues from images and/or LiDAR for 3D reconstruction (Fig. 2).
We ﬁrst describe our model representation and how it leverages CAD priors (Sec. 3.1).
Then we present our energy-based model (Sec. 3.2), and how we conduct inference to generate the vehicle mesh (Sec. 3.3).
1 Model Representation
To faithfully simulate an object, we need to recover not only its geometry, but also its material.
Towards this goal, we exploit meshes as our underlying representation since they best meet the desired properties of modern high-ﬁdelity simulators, by being efﬁcient to animate, edit, and render.
Mesh representation.
A mesh M = ( V , F ) is composed of a set of vertices V ∈ R | V |× 3 and faces F ∈ N | F |× 3 , where the faces deﬁne the connectivity of the vertices.
Our goal is to deform a mesh to match the observations from the sensor data.
Typically, during deformation, the topology ( i.e ., connectivity) of the mesh is ﬁxed and only the vertices are “moving” [23, 32, 78, 21].
This strategy greatly simpliﬁes the deformation process, yet at the same time constrains its representation power.
For instance, if the original mesh topology is relatively simple ( e.g ., a sphere [78], a “mean-shape” mesh template [32], etc.) and non-homeomorphic to the object of interest, the mesh may struggle to capture the ﬁne-grained geometry ( e.g ., side mirrors of a car).
CAD models as priors.
To circumvent such limitations, we propose to incorporate shape priors from CAD models into the mesh reconstruction.
One straightforward approach is to directly exploit the CAD model to initialize the mesh M CAD =
( V CAD , F CAD ) .
Since CAD models, by design, respect the topology of real-world objects, the mesh may be able to model various details.
One obvious caveat, however, is that there is no structure among the vertices.
Each vertex may move freely in the 3D space during the optimization; and there is no guarantee that vertices belonging to a wheel will continue to form a cylinder.
We thus further incorporate the part information from CAD models into the parameterization.
This allows us to inherently preserve part geometry, as well as enable part manipulation ( e.g ., changing the steering of the wheels).
We leverage the semantic part information from CAD models to partition the vehicle mesh M CAD into a vehicle body { V body , F body } and wheels { V ( k ) wheel , F ( k ) wheel } Ki =1 .
K indicates the number of wheels a vehicle has.
K = 4 for most vehicles but may be 6 or more for large trucks.
We employ the fact that the wheels are usually the same within the same vehicle, and model all wheels with the same underlying mesh { V wheel , F wheel } , composited with their relative pose T k to the vehicle origin.
As there are a wide variety of vehicles with different wheel sizes and different relative positions to the vehicle body, we further add a scale factor r =
[ r w , r h , r w ] (wheel radius and thickness), and per-axle translation offsets t front , t back ∈ R 3 with respect to the wheel origin.
We can thus drastically reduce the degrees of freedom, enforce the shape of the wheel, and guarantee the relative positions of the wheels to be symmetric.
Importantly, the front-axle wheels can be steered and do not necessarily align with the body.
We hence parameterize the front wheels to have a yaw-relative orientation ρ .
The full vehicle mesh model (see Figure 3) can thus be written as:
[Equation]
with V body , r , t back , t front , ρ being the free variables.
Extending to a CAD library.
While a CAD model provides a useful prior for mesh initialization, it is likely that a single template mesh may not sufﬁciently cover a wide range of objects.
We thus resort to a CAD model library that consists of various vehicle types.
Inspired by [31, 76], we represent a large variety of vehicles with a compact, low dimensional code, which allows us to quickly determine the best CAD model to initialize from.
We ﬁrst parameterize a collection of the CAD models with
Eq.
(1), simplify them so that they have the same number of vertices and topology, and align them in a shared coordinate space with dense correspondences.
Finally, we apply principal component analysis on vertex coordinates to obtain a shared low-dimensional code z .
Note that since all CAD models are parameterized with Eq.
(1), the deformed mesh will reﬂect this part structure as well.
Please refer to the supp.
material for more details.
Appearance representation.
In addition to geometry, our mesh representation must accurately capture how light interacts with its surface so that we can realistically reproduce sensor observations.
Towards this goal, we parameterize the mesh appearance using a physically-based appearance representation.
Speciﬁcally, we represent appearance using a micro-facet BRDF model with the differentiable split sum environment lighting [79] proposed by Munkberg et al.
Since the topology of the mesh is ﬁxed, we can build a one-to-one mapping relationship between each point on the mesh surface and each point in the 2D ( u, v ) space.
We use a physically-based (PBR) material model from Disney [80], which contains a diffuse lobe k d with an isotropic, specular GGX lobe [81].
3.2 Energy Formulation
The next step after the CAD model initialization is to optimize both the geometry and the appearance of the mesh such that it best matches the captured sensor data.
Our approach builds upon recent success on differentiable rendering [82, 83, 84, 85], leveraging a differentiable renderer that takes as input variables like the sensor pose and our textured mesh representation, and outputs a realistic simulation of the object.
Differentiable rendering allows all variables to be optimized based on the supervision provided by the input images.
Notation.
Let I =
{ I i } 1 ≤ i ≤ N be the images captured at different timestamps and P be the aggregated LiDAR point clouds captured by a data collection platform, in our case a self-driving vehicle driving in the real-world.
Let { M i } 1 ≤ i ≤ N be the foreground segmentation mask of { I i } 1 ≤ i ≤ N obtained from a off-the-shelf algorithm [30, 86].
Let A =
{D , R} be the variables directly related to the appearance model ( i.e ., the material and the lighting).
We denote Π
= { K cam i , ξ cam i , ξ lidar } as the intrinsics and the extrinsics of the sensors, where ξ ∈ se (3) is the Lie-algebra associated with SE(3).
We assume all cameras are pre-calibrated with known intrinsics.
Let ψ : ( M , A , Π) →
( I ψ , M ψ ) be the differentiable renderer where I ψ and M ψ denote the rendered RGB image and object mask.
Overall energy.
We design an energy function with complementary terms which measure the geometry and appearance agreement between the observations and estimations ( E data ), while regularizing the shape ( E shape ) and appearance ( E app ) to obey known priors:
[Equation]
We now describe in more detail each energy term.
Data term.
This energy encourages the estimated textured mesh to match the sensor data as much as possible.
Speciﬁcally, it consists of three components:
[Equation]
E color , E mask both enforce the rendering of the textured mesh to be close to the image observations.
They, however, are complementary.
E color encourages the appearance of the rendered image to match the RGB observation and propagates the gradients to all variables including appearance variables
A , while E mask measures the mask difference and only depends on the shape of the mesh as well as the camera poses.
Following previous work [30, 67], we exploit smooth- (cid:96) 1 distance to measure the difference in RGB space and squared (cid:96) 2 for object masks:
[Equation]
with ¯ (cid:96) 1 the smooth- (cid:96) 1 norm [87] and N the number of images available.
E lidar encourages the geometry of our mesh to match the aggregated LiDAR point clouds.
Since minimizing point-to-surface distance is computationally expensive in practice, we adopt the popular
Chamfer Distance (CD) to measure the similarity instead.
We randomly select L points ( P s ) from the current mesh and compute the asymmetric CD of P s with respect to the aggregated point cloud P :
[Equation]
where α is an indicator function representing which LiDAR point is an outlier.
We refer the reader to
Sec.
3.3 for a discussion on how to estimate α .
Shape term.
This energy encourages the deformed mesh to be smooth and the faces of the mesh to be uniformly distributed among the surfaces (so that the appearance would be less likely distorted).
Speciﬁcally, E shape is an addition of two shape regularizations [85, 63, 30]: a normal consistency term E normal ( V ) and an average edge length term E edge ( V ) :
[Equation]
Here, f ∈ F and v ∈ V refer to a single face and a single vertex.
N ( f ) and N ( v ) denote the neighboring faces of f and the neighboring vertices of v respectively.
N F and N E are the number of neighboring face pairs and edges.
Appearance term.
This energy exploits the facts: (1) the appearance of a vehicle will not change abruptly in most cases; instead, it varies in a smooth fashion; (2) it is primarily neutral, white lighting in the real world.
Following prior art in intrinsic decomposition [88, 89], we adopt a sparsity term to penalize frequent color changes on the diffuse k d and specular k s terms.
We also add a regularizer to penalize the environment light in gray scale.
[Equation]
where ∇ k d and ∇ k s are the image gradients approximated by Sobel-Feldman operator [90].
c i and ¯ c i are the light intensity values at R,G,B channels and the per-channel average intensities.
3.3 Inference
Our goal is to ﬁnd the optimal mesh M ∗ , appearance representation A ∗ , and sensor poses Π ∗ such that the total energy is minimized.
We adopt the following strategy for better inference convergence.
Initialization.
A good initialization is required for non-convex optimization based methods to achieve good performance.
If the energy model is initialized from CAD models that are very different from the observations, the mesh may not be able to ﬁt the sensor data well.
To handle a wide variety of vehicles, we leverage the learned low dimensional latent code z to estimate the initial coarse mesh
M init =
( V init , F ) , where V init is reconstructed from the optimized latent code z ∗ and
[Equation]
We note that we focus on the geometry and only optimize the latent code.
The latent code is initialized from 0 , and the sensor poses are obtained from coarse calibration and ﬁxed.
We optimize z using stochastic gradient descent with Adam optimizer [91].
Please see supp.
material for additional inference details.
Since the low dimensional latent code already smooths out instance-speciﬁc details, the optimization is much more robust and less sensitive to initialization.
Efﬁcient and robust inference.
Given the initialization M init , we jointly optimize the vertices
V , appearance variables A , and sensor poses Π as described in Eq.
To save computation, we uniformly sample L points on the current mesh at each iteration to compute the LiDAR energy
E lidar = CD( P , P s ) , where P s ∼ M and |P| =
L .
To handle LiDAR outliers, we estimate the indicator function α by only calculating the Chamfer distance for the top p percentage of point pairs with the smallest distance.
We ﬁnd it is sufﬁcient to handle point outliers in most cases.
For the sensor extrinsics, we optimize the 6D rotation representation [92] and 3D translation.
It is a well known issue that optimizing vertex positions with adverse gradient descent steps can cause self-intersections that is irrecoverable or even exacerbated in further steps [63, 33].
Therefore, following Nicolet et al.
[33], we use preconditioned gradient descent steps to bias towards smooth solutions at each iterations, which enables faster convergence and higher mesh quality ( i.e ., more ﬁne-grained details).
Experiments 4.1 Experimental Setup
Datasets.
We evaluate our approach on two in-the-wild datasets: Multi-view Marketplace Cars (MVMC) dataset [30] and PandaVehicle dataset [93].
MVMC consists of a diverse set of vehicles captured under various illumination conditions.
Each vehicle comes with around 10 views and rough camera pose.
Following [30], we evaluate our model as well as the baselines on the selectedvehicles.
PandaVehicle is a dataset we derived from PandaSet [93].
It consists of 200+ vehicles, along with their corresponding multi-view images and aggregated LiDAR point clouds.
We manually inspect and select 10 high-quality vehicles for evaluation based on the accuracy of LiDAR-camera calibration.
On average, each vehicle is captured from 24 camera views.
Baselines.
We compare our approach with three types of 3D reconstruction methods: (1) Neural radiance ﬁelds : NeRF++ [43], and Instant-NGP [46].
Implicit surface representations : NeRS [30],
NVDiffRec [67], and NeuS [48].
(3) Explicit geometry-based approaches : single-image view-warping (SI-ViewWarp)
[94], which warps the source image to the novel target view using depth estimated by
LiDAR points; and multi-image view-warping (MI-ViewWarp), which blends multiple source images to the target view.
We also compare against SAMP [31], a CAD model mesh optimization approach which leverages SDF-aligned CAD models for joint pose and shape optimization.
See supp.
material.
Evaluation settings and metrics.
We evaluate reconstruction quality by performing camera and
LiDAR simulation with the reconstructed vehicle.
For camera simulation ( i.e ., novel view synthesis (NVS)), we adopt mean-square error (MSE), peak signal-to-noise ratio (PSNR), SSIM [95],
LPIPS [96] and cleaned FID [97].
To focus on the reconstructed vehicle itself, we only evaluate the foreground pixels using the predicted segmentation mask [86].
For LiDAR simulation, we place the reconstructed vehicle mesh in its original location and perform ray-casting to generate a simulated point cloud.
We compare against held out real LiDAR points.
We evaluate what fraction of real
LiDAR points do not have a corresponding simulated point ( i.e ., coverage), the average per-ray (cid:96)error, Chamfer and Hausdorff distance.
See supp.
for additional details.
4.2 Reconstruction Quality
Novel View Synthesis on PandaVehicle and MVMC.
We report NVS results on PandaVehicle in Tab. 1 and on MVMC in Tab. 2, along with reconstruction time ( T ) and rendering speed ( i.e .,
[Table] [Table]
[Figure]
frame per second (FPS)).
For PandaVehicle we train each approach using images only from left camera, and evaluate on front-left and front cameras.
For MVMC we evaluate each approach via a cross-validation approach, where we hold out one image each time.
CADSim achieves better performance across all image quality metrics, especially LPIPS.
LPIPS measures how neural nets perceive images.
This suggests that our simulation results are perceived by the ML models to be more similar to real-world images (see Sec.
4.3 for more analysis).
[Table] [Table]
With CAD priors, we can reconstruct from much fewer views compared to NeRF-based or implicit-surface based approaches.
Compared to geometry-based warping methods, our differentiable renderer and material model enable much higher photorealism.
Qualitative results on PandaVehicle in Fig.
4 further conﬁrms this.
NeRF-based approaches usually lead to visible artifacts with large view extrapolation and sparse image observations.
Implicit surface approaches perform better by alleviating shape-radiance ambiguity [43], but still have artifacts due to lower quality geometries.
The other geometry-based approaches have better geometry, but have blurry appearance quality or missing pixels compared to CADSim.
To further demonstrate the value of our improved geometry, we compare the NVS results using our reconstructed mesh against a NeRS mesh with various appearance representations in the supp.
material.
Additionally, neural rendering based approaches such as NeuS and NeRF++ are usually slow to train and render.
While Instant-NGP [46] reduces training time signiﬁcantly, it does not support real-time rendering.
LiDAR simulation on PandaVehicle.
We now evaluate the geometry performance of the reconstructed meshes for LiDAR simulation.
As shown in Tab. 3, our approach is better than or comparable to prior art across all metrics, suggesting that our meshes are more complete and more accurate than existing methods.
NVDiffRec has the highest hit rate, yet performs bad on the rest of the metrics.
This is because their generated meshes are noisy, hence covering many LiDAR points.
Ablation study.
To showcase the importance of CAD initalization, we initalize the mesh with the following alternatives: a unit sphere, a rescaled ellipsoid, and geometries from either SAMP [31] or NeRS [30].
As shown in Fig.
5, exploiting CAD priors leads to faster convergence and better performance.
Our reconstruction is able to capture ﬁne-grained details such as wheel size and rotation, rear-view mirrors, etc.
We also evaluate our reﬂectance model choice [81] in the supp.
material.
Our physics-based material and lighting model performs the best and generalizes well to novel views.
(cid:56)(cid:81)(cid:76)(cid:87)(cid:3)(cid:54)(cid:83)(cid:75)(cid:72)(cid:85)(cid:72) (cid:40)(cid:79)(cid:79)(cid:76)(cid:83)(cid:86)(cid:82)(cid:76)(cid:71) (cid:49)(cid:72)(cid:53)(cid:54) (cid:54)(cid:36)(cid:48)(cid:51) (cid:50)(cid:88)(cid:85)(cid:86)
[Figure]
[Figure]
4.3 Realistic and Controllable Simulation
We now showcase applying CADSim for accurate camera simulation to evaluate perception models, and to perform realistic vehicle insertion.
CADSim also supports texture transfer naturally, allowing us to expand simulation assets easily.
Please see additional details and results in supp.
material.
Downstream evaluation on camera simulation.
To verify if CADSim helps reduce domain gap for downstream perception tasks, we evaluate object detection and instance segmentation algorithms [98, 86] on simulated camera images at novel views.
In particular, we consider two setups: rendered and blended [17] and copy-pasted.
We compute the instance-level Intersection over Union (IoU) of predicted bounding box and segmentation mask between simulated image and real images.
This detection/segmentation agreement metric (the larger the better) indicates how well the simulated data align with the real data with respect to these two perception tasks and model architectures.
This metric can provide guidance on using simulation as one tool for evaluation.
As shown in Tab. 4, using
CADSim assets usually leads to the largest agreement with real images under different settings.
Realistic animated vehicle insertion.
Given the original scenario and sensor data and an inserted actor trajectory, we produce consistent multi-sensor simulation with the rendered asset.
In Fig.
6, we generate safety-critical scenarios by inserting a new actor turning into our lane (left half) and an actor slowing down in the adjacent lane, preventing the SDV from lane-changing (right half).
The LiDAR simulation is conducted on reconstructed meshes enriched with intensity values (see supp.
A2).
The rendered asset is seamlessly blended into the original scenario and has physically realistic wheel rotation and movement.
This enables diverse scenario generation for end-to-end autonomy testing.
The original camera and LiDAR observations captured in the real world are at the top-left corners.
Limitations and Conclusion
CADSim’s main assumption is that it requires CAD models for the object class of interest.
We note that CAD models are readily available for most object classes, and that our approach only requires encoding semantic priors for a single CAD asset, as our energy-model optimization allows for transfer of these priors to other assets of the same class.
We demonstrate our approach on other classes ( e.g ., motorcycles and cones) in the supplementary material.
Additionally, our approach relies on either good segmentation masks or LiDAR points to reconstruct accurate shapes.
In this paper, we proposed to leverage in-the-wild camera and LiDAR data to reconstruct objects such as vehicles.
Towards this goal, we designed CADSim, which leverages geometry and semantic cues from CAD models with differentiable rendering, to generate meshes with high quality geometry and appearance.
These cues also enable our approach to generate articulated and editable meshes, enabling endless creation of new shapes, textures, and animations for simulation.
We demonstrated that integrating our meshes into a camera simulation system can more effectively evaluate robot perception algorithms than existing reconstruction methods, bringing smaller domain gap in simulation.
Acknowledgement
We sincerely thank the anonymous reviewers for their insightful suggestions.
We would like to thank Chris Zhang, Yuwen Xiong for their feedback on the early draft and ﬁnal proofreading.
We also thank the Waabi team for their valuable assistance and support