Exploring Adversarial Robustness of Multi-sensor Perception Systems in Self Driving
Abstract: Modern self-driving perception systems have been shown to improveupon processing complementary inputs such as LiDAR with images.
In isolation,2D images have been found to be extremely vulnerable to adversarial attacks.
Yet,there are limited studies on the adversarial robustness of multi-modal models thatfuse LiDAR and image features.
Furthermore, existing works do not considerphysically realizable perturbations that are consistent across the input modalities.
In this paper, we showcase practical susceptibilities of multi-sensor detection byinserting an adversarial object on a host vehicle.
We focus on physically realizableand input-agnostic attacks that are feasible to execute in practice, and show thata single universal adversary can hide different host vehicles from state-of-the-artmulti-modal detectors.
Our experiments demonstrate that successful attacks areprimarily caused by easily corrupted image features.
Furthermore, in modern sensor fusion methods which project image features into 3D, adversarial attacks canexploit the projection process to generate false positives in distant regions in 3D.
Towards more robust multi-modal perception systems, we show that adversarialtraining with feature denoising can boost robustness to such attacks signiﬁcantly.
Introduction
Recent advances in self-driving perception have shown that fusing information from multiple sensors(e.g., camera, LiDAR, radar) [1, 2, 3, 4, 5, 6] leads to superior performance when compared torelying on single sensory inputs.
Such performance gains are primarily due to the complementaryinformation contained in the measurements provided by the different types of sensors.
For example,
LiDAR sensors provide accurate 3D geometry while cameras capture rich appearance information.
Meanwhile, modern perception models which rely on deep neural networks (DNNs) have beenfound to be extremely vulnerable to adversarial attacks when processing images in isolation [7, 8,9, 10, 11].
Adversarial attacks can be thought of as perturbations to the sensory inputs which donot alter the semantic meaning of the scene, but drastically change a DNN’s output and resulting inincorrect predictions.
Such vulnerabilities can lead to catastrophic consequences in safety-criticalapplications.
In the context of self-driving, most efforts have investigated attacks against singlesensor inputs, such as image-only attacks [7, 11] and LiDAR-only attacks [12].
Towards multimodal robustness, [13] considers perturbations of LiDAR and image inputs independently, resultingin perturbations that are inconsistent across modalities and therefore may not be physically realizableand hence not threatening in practice.
On the other hand, some proposed physically realizableapproaches [14] only search over shape but ignore texture which is crucial for corrupting imageinputs.
Furthermore, these prior works do not generate universal perturbations which are perhapsthe most threatening in practice.
Such perturbations are input agnostic and can attack any input in thetraining distribution with high probability, meaning they can be executed without prior knowledge ofthe scene and are able to consistently disrupt models that process sensory information across time.
This paper demonstrates the susceptibility of multi-sensor detection models to physically realizableand input-agnostic adversarial perturbations.
To create a physically realizable attack which is alsofeasible to execute, we focus on object insertion attacks [7, 15, 16, 17, 12], as they can be carried outvia the deployment of physical objects in the real world.
Following [12], we insert the adversarial
[Figure]
object by placing it on the rooftop of a host vehicle.
We render the adversary into LiDAR and imageinputs to ensure perturbations are consistent across modalities and that our attack is physically realizable.
Furthermore, we consider occlusion and environmental lighting in the rendering process asshown in Figure 1 to enhance the realism of simulation.
To enable end-to-end learning of geometryand texture, we render the pixels and LiDAR points in a differentiable manner.
During training, ouradversary is optimized with respect to all vehicles in the training distribution to create a universalattack which can be applied to any vehicle in any scene.
We conduct an empirical evaluation of our proposed attack on the KITTI [18] self-driving datasetand a novel large-scale self-driving dataset X ENITH using the multi-sensor detector MMF [5].
Wegenerate input-agnostic adversarial examples that successfully hide host vehicles from state-of-theart detectors in both datasets.
More importantly, we ﬁnd that incorporating image inputs makesmodels more vulnerable when compared to using LiDAR alone, as successful attacks are primarilycaused by the brittle image features.
Moreover, the projection of image features into 3D allows theadversary to generate false detections in distant regions.
Nonetheless, we show that false negativefailures can be circumvented by applying feature denoising and adversarial training.
However, weobserve that distant false positives are much harder to correct with adversarial defense, as they arealso caused by inaccurate mappings between 2D pixels and 3D LiDAR points during fusion.
Related work
Adversarial attacks were ﬁrst discovered in the 2D image domain, where small perturbations on thepixels were shown to generate drastically different prediction results on object classiﬁcation [19, 20].
Networks trained on object detection and semantic segmentation have also been shown to exhibitsuch vulnerability [8, 9, 21, 22, 23, 24].
Early methods [19, 20, 25, 26] assume the perfect knowledgeof the gradients of the victim model, referred to as whitebox attacks.
Later it was found that ablackbox attack can achieve similar success as well [27, 28, 29].
Defense and robustness evaluationprocedures have also been explored for adversarial attacks [30, 31, 32, 33, 34, 35, 36, 37].
Aside from changing the pixel values by a small amount, various other ways to “perturb” an imagewere also proposed.
Object insertion attacks are realistic attacks that insert an object to changethe network output while not introducing changes in semantics [38, 15, 7, 39, 40].
These attackswere originally designed to be stickers that can be attached to a target object, and has since alsobeen applied to the lens of a camera [41].
Image rendering is another popular technique for nonpixel based attacks, which can also be made differentiable [42], by using which [43] showed thatadversarial attacks can be made through changing lighting.
Various other object insertion attacksdesigned camouﬂage textures that can be wrapped around the target object [44, 10, 45, 46, 47, 48].
Aside from the typical image-based attacks introduced above, adversarial attacks against pointclouds have also been studied.
[17, 49, 50, 51] tried to directly perturb the location and cardinality of point clouds.
However, such attacks may not be physically realizable, as arbitrary perturbations won’t be achievable by a LiDAR sensor with ﬁxed angular frequency.
Towards morerealistic attacks, [52, 53] developed spooﬁng attacks that add malicious LiDAR points, while otherapproaches [16, 54, 12] instead optimize 3D mesh surfaces and use differentiable ray-casting togenerate LiDAR point clouds.
[Figure]
Despite the fact that multi-modal sensor conﬁgurations are widely seen on self-driving vehicles [55,5, 56, 57, 58, 59], research on multi-modal sensor attacks is still very limited.
Several preliminaryworks show the possibility of attacking multi-sensor fusion networks [13, 14, 60].
However, [13]did not consider consistency across data modalities when perturbing the image input, whereas [14]did not consider image texture, resulting in a lack of attack expressivity, and [60] did neither.
Webelieve that it would be an interesting question to ask, whether multi-sensor fusion can be mademore robust when the attacks are both input-agnostic and physically realizable.
Multi-sensor Adversarial Learning
In this section, we present a general method for learning an adversarial textured mesh to attack anymulti-sensor object detector that is differentiable end-to-end.
Speciﬁcally, we require the adversarialattack to be (1) input-agnostic for different environments, (2) geometrically-consistent across imageand LiDAR input modalities, and (3) fully-automatic for implementation at large-scale.
Our attacksare focused on vehicles as they are the most common object of interest on the road.
Preliminaries: We consider a bird’s eye view (BEV) object detection model F that takes the frontcamera image x I ∈ [0 , 1] H × W ×
and LiDAR point clouds x L ∈ R P × 3 as input x = ( x I , x L ) .
Here,the dimensions H and W represent the image height and width respectively.
P is the number of
LiDAR points which could vary in each frame.
The object detector is trained on bird’s eye view(BEV) bounding box annotations Y , with each bounding box instance b ∈ Y parameterized byb =
( b x , b y , b h , b w , b α ) .
Subsequently, b x and b y are coordinates of the bounding box center, b h andb w indicate the width and height, respectively, and b α represents the orientation.
In order to processboth image and LiDAR data modalities, the object detector uses two separate branches to extractfeatures from each modality (see Fig 2).
Then, the 2D image features are projected into 3D space tobe fused with the LiDAR features.
3.1 Multi-sensor Simulation for Object Insertion
In this work, we design a framework to insert a textured mesh into the scene so that both appearanceand shape can be perturbed to attack multi-sensor perception systems.
We attach a triangle mesh
M =
( V , F , T )
onto the roof of a host vehicle, as such placement is physically realizable in thereal world.
The mesh is parameterized by vertex coordinates V ∈ R N × 3 , vertex indices of faces
F ∈ N M × 3 , and per-face vertex textures T ∈ R M × C × C × 3 .
The dimensions N , M and C representthe number of vertices, the number of triangle faces, and the per-face texture resolution, respectively.
For scalability reasons, we do not consider transparency, reﬂective materials, or shadowing, as handling each case would require sophisticated physics-based rendering.
Instead, we approximate thesensor simulation using LiDAR ray-tracing and a light-weight differentiable image renderer.
Bothimage and LiDAR rendering pipelines are differentiable, allowing gradients from LiDAR points andimage pixels to ﬂow into the mesh parameters during optimization.
The overall pipeline of for objectinsertion is illustrated in Figure 1.
Rooftop Approximation: First, we estimate the center of the vehicle’s rooftop to determine the 3Dlocation for placing the adversary.
Following [61, 12, 62], we represent our vehicle objects usingsigned distance functions (SDFs) and further project them onto a low-dimensional shape manifoldusing PCA.
For each vehicle, we then optimize the low-dimension latent code that minimizes theﬁtting error between the vehicle point clouds and the shape manifold.
Then, a ﬁtted SDF is convertedto a watertight vehicle mesh with Marching Cubes [63].
We select the top 20cm of the vehicle asthe approximate rooftop and use the rooftop center and vehicle heading to determine the exact posefor object insertion.
LiDAR Simulation: To simulate insertion in the LiDAR sweep, we sample rays according to LiDAR sensor speciﬁcations used to collect the original sweep, such as the number of beams andthe horizontal rotation rate.
We then compute the intersection of these rays and mesh faces usingthe Moller-Trumbore algorithm [64, 12] to obtain a simulated point cloud of the adversarial mesh.
These simulated points are then added to the original LiDAR sweep.
Image Simulation: To render the adversary into the image, we extract the intrinsics and extrinsicsfrom the camera sensor that captured the original image.
We then use a light-weight differentiablerenderer SoftRas [65] to simulate the mesh pixels.
Using a soft rasterizer during optimization allowsgradient ﬂow from occluded and far-range vertices to enable better 3D reasoning from pixel gradients.
To enhance the ﬁdelity of rendered images, we model the sun light with a directional lightingmodel as a light at inﬁnite distance.
Occlusion Reasoning:
As we insert a new object into the scene, the rendering process must alsoconsider occlusion for both the original and newly rendered points and pixels.
To handle LiDARocclusions, we compare the depth of existing points in the LiDAR sweep and newly rendered points,discarding the point that is farther away.
Unlike LiDAR where the depth of each point is known,raw images do not contain depth information.
To obtain depth estimates, ﬁrst the LiDAR pointsare projected onto the image to generate a sparse depth image since images have higher resolutionthan LiDAR.
We then use a depth completion model [66], which uses the sparse depths and RGBimage to compute dense per-pixel depth estimates.
With depth estimates of each pixel, we overlayrendered pixels onto the original image and discard occluded pixels.
Note that we do not attack thedepth completion model as it is a preprocessing step used to increase the ﬁdelity of simulation.
3.2 Universal Adversarial Example Generation
Adversarial Objectives: We consider two adversarial objectives: one for false negatives and theother for false positives.
To generate false negative attacks, we follow prior work [8, 12] in attackingobject detectors by suppressing all relevant bounding box proposals for the host vehicle.
A proposalis relevant if its conﬁdence score is greater than 0.
and it overlaps with the ground-truth boundingbox.
The adversarial loss then minimizes the conﬁdence of all candidates:
[Equation]
where IoU denotes the intersection over union operator and b ∗ is the corresponding ground-truthbox we aim to attack.
Alternatively, we aim to generate false bounding box proposals that do not overlap with any groundtruth boxes in the scene.
The false positive adversarial loss increases the conﬁdence of the falsepositive candidates as follows:
[Equation]
where (cid:101)
Y fp is a subset of bounding box proposals with no overlap with any ground-truth objectbounding boxes.
Mesh Regularization: Besides the adversarial objective, we use an additional regularization termto encourage realism in the mesh geometry.
Speciﬁcally, we use a mesh Laplacian regularizer [65],which encourages smooth object surface geometries: L lap = (cid:80) i (cid:107) δ i (cid:107) 22 , with δ i as the distance fromvertex v i ∈ V to the centroid of its immediate neighbors N ( i ) :
δ i =
v i − 1 (cid:107)N ( i ) (cid:107) (cid:80) j ∈ N ( i ) v j .
Inaddition to Laplacian regularization, we also constrain the physical scale of the adversary with an
[Figure]
[Figure]
IoU
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
XENITH
[Figure]
1.0 IoU 0.
KITTI
[Figure]
axis-aligned 3D box.
Namely, we require that (cid:107)V j (cid:107) ∞ ≤ L j for j ∈ { x, y, z } , where L x , L y , and
L z represent the box constraints along xyz -axis, respectively.
Learning Input-Agnostic Attacks: Overall, our optimization objective can be summarized as L =
L fnadv + λ fp L fpadv + λ lap L lap , where λ fp and λ lap are coefﬁcient that weight the relative importance ofthe false positive loss term and mesh regularization term.
We employ this objective to optimizethe shape and appearance of the inserted object on the entire dataset to generate an input-agnosticadversarial example.
Therefore, we can denote the optimal adversary as the following equation:
[Equation]
[Equation]
With our proposed pipeline which is differentiable end-to-end, optimization of the adversarial meshis done using projected gradient descent to respect the (cid:96) ∞ constraints on mesh vertices.
In ourexperiments, we also conduct attacks targeting a single modality.
To achieve this, we disable thegradient ﬂow to the untargeted input branch, while we still simulate the mesh into both modalitiesto maintain physical consistency across image and LiDAR modalities.
3.3 Multi-sensor Adversarial Robustness
Towards defenses against our object insertion attack, we also study defense mechanisms.
Comparedto the single-sensor setting, achieving multi-sensor adversarial robustness is even more challenging.
First, each single input modality could be attacked even when the perturbations on the other inputsensors are non-adversarial.
Second, adversarial perturbations from each single input modality caninteract with each other, which is a unique aspect in the multi-sensor setting.
Thus, we need to dealwith not only perturbations at each input modality but also their effect in the fusion layer.
We employ adversarial training as it is the most standard and reliable approach to defense.
Adversarial training can be formulated as solving for model parameters
[Equation]
which minimize the empirical risk under perturbation
Here L task is the loss function used to train thedetection model.
This is achieved by training detection model F against adversarial data (cid:101) x generatedby our attack.
While adversarial training is typically performed on image perturbations that arecheap to generate with only few PGD steps [67], our adversarial example generation is prohibitivelyexpensive for the inner loop of the min-max objective.
Thus, instead of generating a strong adversaryfrom scratch at every iteration, we adopt free adversarial training [68] and continuously update thesame adversary to reduce computation.
Experimental Evaluations
In this section, we ﬁrst describe the our datasets, attack protocols, and evaluation metrics.
More details on experiment setting are provided in the supplementary material. We then conduct experiments
[Figure]
KITTI
XENITH on the multi-sensor detection model MMF [5] and present our empirical ﬁndings for white-box attacks on each dataset and the black-box transfer attacks across datasets.
Finally, we explore severaldefense mechanisms to achieve a more robust multi-sensor object detector.
4.1 Experimental Setting
Datasets: We conduct our experiments on two self-driving datasets: KITTI [18] and X ENITH .
X ENITH is collected by having a ﬂeet of self-driving cars drive around cities in North America.
Snippets are captured in the daytime and detection is performed on objects within 70 meters forwardand 40 meters to the left and right of the ego-car.
Each vehicle in a frame is considered a separatesample and we have 12,284 vehicles in KITTI and 77,818 in X ENITH .
Metrics: Following prior work on object insertion attacks [12], we evaluate how often the hostvehicle “disappears” by measuring its recall across various IoU thresholds.
For a scalar metric, weevaluate the false negative attack success rate (FN ASR) as the percentage of host vehicles detectedbefore perturbation that are undetected after perturbation.
We consider a vehicle detected if thereexists an output bounding box having greater than 0.7 IoU with the vehicle.
On the other hand, weconsider an output bounding box a false positive if its maximum IoU with any ground truth boxis less 0.3 and it does not overlap with any detection produced before perturbation.
We evaluatethe false positive attack success rate (FP ASR) as the percentage of attacks which generate at leastone false positive.
Finally, the overall attack success rate (ASR) is the percentage of attacks whichsuccessfully create a false positive or false negative.
Implementation Details: The adversarial mesh is initialized as an icosphere with N = 162 vertices and M = 320 faces.
Per-face textures are parameterized using a texture atlas with a 5 × 5texture resolution for each face.
During optimization, we set λ lap = 0 .
001 , λ fp = 1 , and further constrain the scale of the mesh with an axis-aligned 3D box where the x and y coordinates are boundedby L x =
L y = 0 .
8 m and the z coordinate is bounded by L z = 0 .
5 m .
We use Adam [69] to optimize the mesh parameters with a learning rate of 0 . 004 for textures and 0 .
001 for vertex coordinates.
To target either LiDAR or image branch in isolation, we disable gradient ﬂow to the other branchduring the backward pass to the adversary.
4.2 Universal Adversarial Attacks
Hiding Host Vehicle:
We evaluate the drop in recall in detecting the host vehicle, as missed detections can lead to colliding with unseen objects which is the most dangerous outcome.
We sweep
IoU thresholds and visualize the IoU-recall curve in Figure 3.
First, inserting a mesh with randomized shape and appearance has little impact on the detector.
On the other hand, an adversarial meshgenerated by perturbing both input modalities leads to a signiﬁcant drop in recall.
Moreover, weperturb the LiDAR and image inputs in isolation and ﬁnd that targeting the LiDAR inputs alone [Table] [Table]
[Figure] yields very weak attacks. is almost both modalities.
Therefore, image inputs are signiﬁcantly less robust to the proposed [Figure]
Attack Success Rates:
In Table 1 and Table 2, wefurther analyze results in terms of the attack successrates.
Again, we consider meshes with randomlygenerated geometry and texture as a baseline.
Weobserve similar trends of image features being signiﬁcantly more vulnerable.
In addition to misseddetections, the adversarial mesh is able to attack thedetector through generating false positive proposalsof objects that do not exist.
Furthermore, we compare against prior work [12] which attacks a LiDARonly detector.
In this case, incorporating image inputs boosts robustness to LiDAR attacks at the costof being more vulnerable to multimodal attacks. Adversary Size Furthermore, to understand howmuch the size of the adversary affects the strengthof the attack, we vary the size of the box constraints on the vertex parameters. Here we sweep
L x = L y = L z = L and and measure the attack success rates.
Results are shown in Figure 5. Asexpected, the attack becomes stronger as the (cid:96) ∞ constraints on vertex coordinates are relaxed. Qualitative Examples: Qualitative examples are shown in Figure 4.
First, the detector fails todetect host vehicles with the adversarial mesh on its rooftop.
We show detections in the image ratherthan LiDAR for ease of viewing.
Note that the same adversarial mesh is used for all demonstrations,as the attack is agnostic to the host vehicle and environment.
Furthermore, we show in Figure 6that our adversarial mesh generates false positives at very distant locations.
Here, detections arevisualized in BEV since distant objects appear too small in the image.
Additionally, we visualizeimage features in the image plane and the visual cone of projected image features into 3D, showingthat long-range false positives are caused by strong image features dominating after fusion.
Black-box Transfer Attacks: We conduct transfer attacks across datasets and show results in
Table 3.
Overall, our transfer attack on the target dataset is stronger than attacking only the LiDARinput modality on the source dataset, especially from X ENITH to KITTI.
On the other hand, thetransferability is probably lowered by the image resolution and hardware, which is beyond the scopeof our paper but an interesting future direction to explore.
[Table] [Table]
Attacks Against Defense Methods: As empirical ﬁndings suggest that the image feature is morevulnerable, we ﬁrst employ an existing image-based defense method that removes high-frequencycomponent through JPEG compression [70].
In addition, we conduct adversarial training against theattacker.
Since generating a strong adversary is extremely expensive due to the simulation pipeline,we employ a strategy similar to Free Adversarial Training [68] and reuse past perturbations by continuously updating the same adversarial object.
Speciﬁcally, we perform 5 updates to the adversaryper one update to the model.
We combine the feature denoising [36] with the adversarial training tofurther enhance robustness against image perturbations in particular.
We report the success rates aswell as the average precision (AP) at 0.7 IoU to study the trade-off between adversarial robustnessand performance on benign data [35].
As shown in Table 4, we ﬁnd JPEG compression is very ineffective as defense.
We hypothesizethis is because the input-agnostic adversary is rendered at various different poses during trainingand therefore do not rely on high-frequency signals that are removed by JPEG compression.
Incomparison, our adversarial training effectively reduces the overall attack success rate from 49.76%to 14.97%, while dropping AP by 0.5%.
Finally, adding non-local mean blocks after every residualblock in the image processing backbone further improves robustness by reducing the ASR by 5%.
Discussions and Future Work: While adversarial training methods are effective, they are speciﬁcto a speciﬁc threat model and may not generalize to unseen perturbations.
A more threat-agnosticmechanism like more robust sensor fusion would bring more robustness in general.
Furthermore,adversarial defense methods are only effective at recovering the missed detections, but struggle todetect false positives.
We believe this is because distant false positives shown in Figure 6 are onlypartially due to vulnerabilities to adversarial perturbations.
In fact, such examples exploit erroneousassociations between objects that are distant in 3D. Speciﬁcally, the mapping between a mesh pixeland a LiDAR point far away from the mesh enables such attacks.
These false associations caneasily occur if the assigned pixel for each LiDAR points is shifted by a few pixels, since objectswhich are far apart in 3D may appear very close in 2D. We identify two reasons how this canoccur in practice.
First, due to the receptive ﬁeld of DNN activations, an adversarial object caninﬂuence pixels outside its physical boundaries.
Second, while LiDAR sweeps are collected in acontinuous fashion with a rotating sensor, images are captured instantaneously at regular intervals.
Consequently, the camera extrinsics used for projection become outdated for LIDAR points capturedbefore and after the image.
Thus, to achieve more robust sensor fusion for images and LiDAR,fusion modules must reason about 3D geometry, contextual information, and temporal informationof LiDAR points to generate mappings between image pixels and LiDAR points more intelligently.
We hope these ﬁndings will inspire future work towards more robust sensor fusion methods.
Conclusions
Our work invetigates practical adversarial attack against mulit-sensor detection models in selfdriving to better understand how consuming different input modalities affectss adversarial robustness.
Compared to existing attacks against multimodal detectors, our object insertion attack is morethreatening in practice as we generate input-agnostic and physically realizable adversarial perturbations.
The proposed method is capable of hiding existing vehicles and generating false detections at the same time.
Our experiments reveal that vulnerabilities of multi -sensor object detectors areprimarily due to non-robust image features.
While adversarial training can effectively recover the missed detections, we ﬁnd it still struggles to detect false positives without a deeper reasoning about 3D geometry in feature fusion.
We believe this work would open up new research opportunities andchallenges in the ﬁeld of multi-sensor robustness