XC: Exploring Quantitative Use Cases for Explanations in 3D Object Detection
Abstract
Explainable AI (XAI) methods are frequently applied to obtain qualitative insights about deep models’ predictions.
However, such insights need to be interpreted by a human observer to be useful.
In this paper, we aim to use explanations directly to make decisions without human observers.
We adopt two gradient-based explanation methods, Integrated Gradients (IG) and backprop, for the task of 3D object detection.
Then, we propose a set of quantitative measures, named
Explanation Concentration (XC) scores, that can be used for downstream tasks.
These scores quantify the concentration of attributions within the boundaries of detected objects.
We evaluate the effectiveness of XC scores via the task of distinguishing true positive (TP) and false positive (FP) detected objects in the
KITTI and Waymo datasets.
The results demonstrate improvement of more than 100% on both datasets compared to other heuristics such as random guesses and number of LiDAR points in bounding box, raising conﬁdence in XC’s potential for application in more use cases.
Our results also indicate that computationally expensive XAI methods like IG may not be more valuable when used quantitatively compare to simpler methods.
Introduction
Recent development in deep neural networks (DNNs) has led to the state of the art performance on 2D [8, 9, 16, 21, 22] and 3D [12, 27, 37] object detection tasks.
However, despite of the improvement in model performance, the lack of model interpretability remains a signiﬁcant drawback.
This issue has sparked interest in explainable artiﬁcial intelligence (XAI).
The primary goal of these XAI methods is to uncover the logic behind the models’ decisions [14].
Several recent methods have been proposed to generate interpretable visual explanations [28, 36, 30, 25, 32].
They are usually analyzed by a human observer to understand the model’s reasoning for some particular decisions.
Hence, explanations are very useful for debugging and diagnosis.
However, it would be impossible for a human to analyze each instance in a large dataset.
To analyze explanations on a large scale, it is necessary to derive quantitative measures from the explanations.
There have been a few works in this direction [24, 6, 23], but the input data experimented on are 2D image, text, or tabular data.
We have not found any experiments on explanations-related quantitative measures applied to LiDAR point cloud input or to the task of object detection.
Our motivation is to ﬁll in the gap by exploring quantitative usage of explanations for LiDAR-based object detection.
We study explanations in the form of attribution maps for a 3D object detector named PointPillars [12] and use them for the problem of distinguishing true positive (TP) vs. false positive (FP) predictions.
Attributions, in the context of XAI, denote the inﬂuence of input features on model output.
We generate attribution maps using two XAI methods: 1) Integrated Gradients (IG) [32], selected for its axiomatic properties; 2) backpropagation [28], selected for its low computational cost.
Our main contributions are 1 :
• We demonstrate that quantitative usage of explanation for 3D object detection is a promising direction for future research: We propose a set of quantitative metrics called Explanation Concentration (XC), which measures the concentration of attributions within each predicted object.
XC can be used to classify the TP vs. FP predictions effectively, often achieving more than 100% improvement compared to simple heuristics such as number of LiDAR points within a predicted box.
• We discover that XC scores derived from backpropagation can perform better than those derived from IG.
• We propose a new score that can identify TP vs. FP pedestrian predictions better than the individual XC scores and object class score.
This score is generated by combining the XC scores with object class score using a MLP.
Related work
Explanation methods A simple form of explanation is an input feature saliency map obtained via backpropagation: one computes the partial derivatives of the model output with respect to the input features [28].
These partial derivatives are a measure of importance (also called attributions ) for the input features.
Deconvolution [36] and Guided-backprop [30] are similar methods.
Integrated
Gradients (IG) [32] is another gradient-based explanation method that computes multiple inputs along a straight-line path from a baseline input to the original input through afﬁne transformations.
IG then combines gradients from all these inputs to get feature attributions.
The baseline input is deﬁned as an input which generates zero output.
In the case of image input, the baseline is deﬁned as a black image with pixel values all equals to zero.
Sundararajan et al.
[32] demonstrate that IG satisﬁes several axiomatic properties for explanations such as sensitivity (attribution values for nonzero features are nonzero, and attribution values for zero-valued features are zero) and
completeness (sum of attributions equals to model output value).
Quantitative usage of explanations Several attempts have been made to utilize explanations quantitatively.
Ross et al.
[24] use the gradient of the log of model output with respect to input features as attributions.
They design a loss term using prior knowledge about feature relevance to punish the model for having attributions for irrelevant features, hence forcing the model to be "right for the right reasons" by focusing on the relevant features only. Rieger et al. [23] and Du et al. [5] also propose similar methods to constrain feature attribution values.
Erion et al.
[6] realize that such prior knowledge about input feature relevance would not always be available, hence they propose a new loss named “pixel attribution prior" (PAP) which only aims to reduce variations in attribution values for neighbouring pixels.
Their reasoning is that a robust model should not be heavily inﬂuenced by small variations in a few disconnected pixels; rather, it should learn patterns from patches of pixels and neighbouring pixels should have similar attributions.
LiDAR-based object detection A popular technique for point cloud-based object detection is to ﬁrst divide the 3D point cloud into grids called voxels, learn features for each voxel, then apply convolutional layers to the voxel-wise features to extract higher level features.
Techniques used for 2D object detection, such as region proposals and anchor boxes, can then be applied to the extracted feature maps.
VoxelNet [37], SeCOND [35], and PointPillars [12] all partly adopted this strategy
PointPillars’ fast inference speed (62 Hz) makes it a desirable choice for deployment on embedded hardware, such as in autonomous vehicles.
False positive detection Several studies have been conducted on identifying false positive (FP) predictions.
Hendrycks and Gimpel [10] demonstrated that the softmax class probability could effectively distinguish true positive (TP) and false positive (FP) predictions in multiple datasets, with performance far exceeding random guess.
Chen et al.
[2] also used class score to ﬁlter out FP pedestrians predictions iteratively.
Methods designed for identifying adversarial attacks or detecting out-of-distribution (OOD) samples may also be applied to detect FP samples.
Some notable works are feature squeezing [34], LID [18], GraN [17], ODIN [15] and Lee et al.
Explanation in the form of attributions
As mentioned before in Section 2, explanation can be presented in the form of input feature attributions, where the magnitude of the attributions corresponds to feature importance, and the sign of attributions indicates positive or negative inﬂuence on the model output.
In our approach, attributions are generated by IG [32] and backprop [28] for PointPillars [12].
PointPillars has three parts as shown in Fig.
1: a pillar feature net which learns a 2D bird’s eye view (BEV) voxelized feature map called the psuedo image from the original point cloud, a backbone 2D CNN to extract more features from the psuedo image, and SSD [16] as detection head to generate 3D object predictions.
IG’s effectiveness on image data is well-recognized.
However, there are a ﬁxed number of pixels at ﬁxed locations in an image; whereas in a point cloud, both the quantity and location of the points can vary greatly.
IG’s input transformations may not be meaningful for point cloud.
Hence, we chose to generate attributions using the pseudo image as input, so that IG could be directly applied.
Note that our pipeline in Fig.
1 is not limited to IG or backprop explanations only, any other XAI method that produces explanations in the form of feature attributions could be applied too.
There could be many predictions in the same pseudo image.
To explain the different predictions separately, each attribution map is generated for a speciﬁc predicted box and its class label.
In Fig.
2, the IG attributions generated for a car prediction is shown.
For this particular example, the positive attributions highlight features that increase the model’s belief that the object is a car, whereas the negative attributions reduce this belief.
Explanation concentration (XC)
We propose a new set of scores called Explanation Concentration (XC) which measures the concentration of attributions within a given predicted object’s boundaries.
As shown in Fig.
2, many pixels outside of the object have noticeable attributions too, indicating that context features can inﬂuence model output as well.
The XC scores are partly motivated by previous studies which demonstrate that overly-relying on context can hurt model performance for both image classiﬁcation [26] and 3D object detection [27].
The process of computing the XC scores is as follows.
First, denote the i th predicted box in a pseudo image as Pred i .
Then denote the sum of positive attributions across all channels in the pixel at location ( x, y ) of the attribution map as a +( x,y ) .
The idea is to avoid having positive and negative channel values cancelling each other out at a speciﬁc pixel.
Using thresholds to eliminate noisy signals is common practice in computer vision.
For example, in the Canny edge detection algorithm [1], thresholds are used to mask out weak edges.
We apply the same idea to the attribution values.
Denote a pixel-wise threshold as a thresh , and use it to ﬁlter out insigniﬁcant attributions (in other words, the sum a +( x,y ) can be called signiﬁcant if it exceeds a thresh ).
An indicator function I +( x,y ) is deﬁned such that it equals to
if a +( x,y ) >
= a thresh ,otherwise.
One way to quantify the concentration of attributions within the predicted box
Pred i is by summing.
Two new variables are deﬁned for each Pred i in a pseudo image:
[Equation]
Now the stage is set for deﬁning an XC score by summing:
[Equation]
XC _ s
+ i is thus the proportion of the positive attributions which lie within Pred i .
The other way to quantify the concentration of attributions for Pred i is by counting.
For this purpose, we count the number of pixels in the pseudo image having signiﬁcant attributions, rather than summing them:
[Equation]
Computing XC by counting might be helpful, because often there are a few outlier pixels with high magnitude attribution values outside of the bounding bounding box (see Fig.
3 for example).
When
XC is computed by counting, these outliers will not have a big effect; but when XC is computed by summing, the values of these outlier pixels can skew the resulting XC value towards the lower end.
One may also compute a similar set of scores by considering the negative attributions only: call them
XC _ s − i and XC _
c − i for a certain box
Pred i .
We observed that pixels located at object boundaries often get labelled as “outside of the box”.
Hence, when calculating any XC scores, the predicted boxes are enlarged by a small margin m on all sides, so that pixels at object boundaries are labeled as inside the predicted box.
Results and discussions 5.1 Evaluation metrics and implementation details
The objective of our experiment is to evaluate XC’s performance on a meta classiﬁcation task: classifying predictions as either FP or TP.
The predicted objects are categorized as TP or FP based on KITTI’s conventions [7].
To evaluate the performance of a speciﬁc score, we could simply apply a score threshold: if the score is above the threshold then the corresponding prediction is TP, otherwise it’s FP.
Then we can evaluate the resulting detection accuracy.
However, the resulting accuracy is a function of the threshold.
To remove the effect of threshold selection and evaluate the performance of different XC scores more fairly, we compute area under the precision recall curve (AUPR) [19] and area under the receiver operating characteristics curve (AUROC) [3], both are threshold-independent performance measures for binary classiﬁcation.
In binary classiﬁcation tasks, typically one class is treated as the “positive” class, whereas the other class is treated as the “negative” class.
We may choose to treat either the TP boxes or the FP boxes as the positive class.
The AUROC metric treats both classes equally and can reﬂect the score’s
(e.g., one of the XC scores) ability in correctly identifying both the positive and negative classes.
On the other hand, the AUPR metric puts more emphasis on the score’s ability to correctly identify the positive class.
For both AURP and AUROC, higher values indicate better performance.
We evaluate the XC scores on three PointPillars [12] models trained on the KITTI dataset [7] and on one PointPillars model trained on the Waymo dataset [31].
We use OpenPCDet’s
[33] implementation of PointPillars for our experiments and adapt their default settings for training.
The ﬁrst three models are trained for 80 epochs on the KITTI dataset, with 3712 frames for training and 3769 frames for validation.
Due to time and resource constraints, we obtain only one more model trained for 30 epochs on 20% of the Waymo dataset, with 31616 frames for training andframes for validation.
To obtain attribution values, we use Captum [29], a model interpretability library developed for PyTorch [20].
We explore both IG [32] and backprop [28] as explanation methods.
For KITTI, the XC scores are obtained on the predicted objects from the 3769 validation frames.
There are on average 67k predicted objects produced by each model.
One pixel in the pseudo image encodes point features in a 0 .
16 m × 0 .
16 m × 4 .
m pillar.
For Waymo, we comptue XC scores for 94k predicted boxes sampled from 800 validation frames.
One pixel in the pseudo image represents point features in a 0 .
33 m × 0 .
33 m × 6 .
m pillar.
For both datasets, we apply m = 0 .
2 m to the predicted boxes and apply a thresh = 0 .
1 to the attribution maps.
5.2 Distribution of XC values
The distribution of XC _ c + values obtained from one of the models trained on KITTI [7] is shown in Fig.
Both the histograms and the empirical cumulative distribution function plots demonstrate that the XC values of TP instances tend to be greater than those of the FP instances.
Another notable observation is that XC values derived from IG are well-dispersed within the range of [0 , 1] for both the TP and FP instances, whereas XC values derived from backprop are mostly below 0.5, with almost all FP instances having XC values below 0.2.
5.3 Using XC to identify TP and FP predictions
The results for TP vs. FP box classiﬁcation using the XC scores on the KITTI dataset [7] are shown in
Table 1.
Each speciﬁc metric in this table is averaged over three models.
The four XC scores derived from IG and backprop are evaluated, along with four other box-wise features (random guess, distance of the predicted box to LiDAR sensor, the number of points inside the predicted box, and the highest class score for the predicted box) serving as baselines for comparison.
Each of the aforementioned features are evaluated by three metrics on different object classes, making up twelve metrics in total.
Note that the object classes in Table 1 (as well as those in other tables in later sections) represent the predicted labels, not the ground truth labels.
Also, in OpenPCDet’s [33] implementation of
PointPillars [12], the class scores are not softmax scores.
Rather, they are class-wise sigmoid scores: a pair of “class vs. not class” scores for each object class.
When computing AUROC and AUPR, the TP boxes are treated as the positive instances, but AUPR_op refers to the AUPR value obtained by treating the FP predicted boxes as positive instances.
In essence,
AUPR reﬂects TP detection performance, whereas AUPR_op reﬂects FP detection performance.
The expected AUROC value for a random score is 0.5, and the expected AUPR value is the proportion of the positive instances.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Pedestrian
[Table]
Referring to Table 1, it is clear that the top class score beats all other features in every evaluation metric.
However, the improvement brought by XC is certainly non-trivial.
One can observe that for all features evaluated, the second best performing feature is most often XC _ c − B .
Although distance to sensor and number of points in predicted box beat the XC scores generated by IG on the three pedestrian class metrics, they are unable to beat any of the XC scores generated by backprop on the pedestrian class.
A very notable case is the AUPR for pedestrian predictions: XC _ c − B achieves 129% improvement compared to number of points and 706% improvement compare to random guess.
The other three XC scores derived from backprop also achieve more than 100% on AUPR for pedestrian compared to the number of points.
Improvements of similar magnitude are also achieved by the backprop XC scores on AUPR for the cyclist class.
These observations indicate that the backprop XC scores are much better at correctly identifying the TP predictions for the pedestrian and cyclist classes than simple heuristics such as number of points.
Another interesting observation is that the XC scores derived by counting usually outperforms those derived by summing.
Such improvement might be due to the outlier attenuation effect mentioned in the second last paragraph of Section 4.
[Table] To ensure that the advantages offered by XC are not speciﬁc to the KITTI dataset [7], we also present results from Waymo dataset [31] in Table 2.
Again, for most evaluation metrics, one of the XC scores is the second best performing feature besides the top class score.
In addition, the XC scores obtained from backprop often show 100% or more improvement on AUPR for pedestrian and cyclist predictions compared to the number of points.
And again, XC _ c − B is the best performing feature in most cases.
Hence, we believe that the advantages of XC are not limited to the KITTI dataset only.
5.4 Why backprop outperforms IG?
[Table] IG [32] is designed to reﬂect feature importance more precisely than other simpler XAI methods such as backprop.
Thus, it would be interesting to know why IGbased XC scores underperform backprop-based XC scores in the task of classifying TP vs. FP predictions, especially for the pedestrian and cyclist predictions (see
AUPR in Table 1 and Table 2).
We suspect that this is due to the difference in XC distribution.
In Fig.
[Table] we present the KS statistic [4] between the distributions of XC values in the TP and FP instances of each object class.
The KS statistic is a measure for goodness of ﬁt between two distributions: greater value indicates greater difference between the two distributions.
Note that the backprop-based XC scores are able to produce much KS statistic between TP and FP distributions in the pedestrian and cyclist class than the IG-based scores.
To alter the distribution of IG-based XC scores, we remove the last step in computing IG attributions.
As mentioned before in Section 2, IG zeros out attributions for input features with zero value.
This is often achieved by multiplying the computed attributions at each input location i by ( x i − x (cid:48) i ) , where x is the input and x (cid:48) is the baseline, which is by default set to zero.
We remove this process and re-calculate IG-derived XC scores on the KITTI dataset [7].
Then we evaluate the new XC scores on the binary classiﬁcation task for TP vs. FP predictions.
The average results of all four IG-derived
XC scores, all four backprop-derived XC scores (these are averaged over the values presented in
Table 1), and of the four modiﬁed IG-derived XC scores are presented in Table 3.
Note that the results are also averaged over all three models trained on KITTI.
The modiﬁed IG XC scores resulted in 115% improvement in AUPR for pedestrian predictions and in 16% improvement in AUPR for cyclist predictions compared to the original IG XC scores.
Improvement can also be observed in AUROC and AUPR_op for the pedestrian and cyclist classes.
The improvements on pedestrian and cyclist objects also coincide with a shift in the distribution of
XC values.
As shown in Fig.
5, the distribution of XC values now appears very similar to that of backprop-based XC values, but very different from that of IG-based XC values.
By making feature attributions proportional to input feature magnitude, IG is able to generate attribution maps that capture salient features in the input, which is one reason why IG attribution maps makes more sense to a human observer compared to a blurry backprop attribution map (interested readers may visit
Sundararajan et al.
[32] for more attribution map examples).
As a result, IG zeros out most of the attributions outside of the bounding box (because the space outside of object bounding boxes are mostly empty, leading to zero input feature values at such locations), and the remaining attributions are mostly concentrated within the bounding box or at its close proximity (see Fig. 2).
Thus, IG is unable to produce mostly very low (< 0.1)
XC values for the FP predictions, leading to signiﬁcant overlap in values TP prediction XC predictions XC scores, making classiﬁcation using XC scores difﬁcult.
These observations echo with Erion et al. ’s
[6] claim that IG’s choice of a zero-valued baseline is problematic.
Take an image of a digit for example, if the background is white but the digit itself is black (i.e., zero), then the zero-valued pixels in fact contains the key features of this image and should not get zero attributions.
Similarly for point cloud inputs, not just the presence of points, but also the absence of points in certain locations can help the model classify the object.
For instance, pedestrian objects are usually ﬁlled with points, whereas car objects have points on its boundaries but are mostly hollow in the middle.
Note that even without multiplying by input values, IG-derived XC scores still cannot beat backpropderived XC scores on the pedestrian and cyclist classes (see Table 3).
Hence, a more expensive method such as IG may produce more visually appealing attribution map, but a less expensive method may have more potential when used quantitatively.
5.5 Combining XC scores with the top class score
In Section 5.3 we demonstrate that the top class score is the best performing box-wise feature in classifying TP vs. FP predictions.
In this subsection, we aim to improve its performance by combining it with XC scores generated from backprop attributions.
We train classiﬁers for different object classes in KITTI [7] and perform an ablation study on the box-wise features.
To conduct the ablation study, we build a new dataset D f from ﬁve features (the top classs score and the four XC scores computed based on backprop attributions) for each predicted box produced by one PointPillars [12] model on the KITTI validation set [7].
Since we have three models trained on
KITTI, we obtain three D f datasets.
For each D f , we ﬁrst group the samples by the predicted object label, then by the number of LiDAR points: those with less than 100 points forms one set, the rest forms another set.
Thus, from one D f , we generate six smaller datasets d f , two for each of the three object classes.
We then build a 2-layer multilayer perceptron (MLP) with PyTorch [20] as a classiﬁer to be trained on d f .
Layer 1 is of size ( d × 3 ) and layer 2 is of size ( 3 × 1 ), where d represents the number of input features per instance.
ReLU activation is applied after the ﬁrst layer, and the sigmoid function is applied after the second layer to obtain an output score.
We train the MLP to classify a predicted box as TP or FP, using binary cross entropy as the loss function.
All input features are normalized based on the following equation prior to being fed into the MLP: z = ( x − µ ) /s , where x is the original feature value, µ is feature mean value, and s is the standard deviation for that feature.
We use the
Adam optimizer [11] with learning rate set to 0.001.
For each experiment, we ﬁrst shufﬂe d f and then augment it by duplicating the instances four times.
Next we apply 5-fold cross validation to the augmented d f , obtaining 5 different 80%/20% train/validation splits.
For the training instances, we also add a small uniformly distributed noise
U ( − 0 .
05 , +0 . 05)
to each feature to help the MLP generalize better.
For each different split, we train the MLP for 12 epochs with batch size = 16 and record three evaluation metrics (AUROC, AUPR, and AUPR_op) of the output score on the validation instances.
We repeat the 5-fold cross validation 5 times, obtaining 5 × 5 = 25 different values for each of the metrics, and record the average.
Note that the above process is repeated for each d f in the three
D f we have, each evaluation metric is averaged over the three D f and shown in Table 4.
In Table 4, when the top class score is the only input feature, we evaluate its performance directly and use it as baseline for comparison; when more than one features are used, we evaluate the performance of the MLP output score.
The most notable observation is that for predictions containing less than 100 points, combining the XC scores with top class score can often result in better performance in distinguishing TP vs. FP predictions, especially among the pedestrian predictions.
The AUPR for pedestrian increased by (0.540 - 0.492) /[-0.
-]
0.492 = 9.8% after combining the 4 XC scores with top class score.
The improvement in AUROC for pedestrian predictions and in AUPR for cyclist predictions also exceed 1%.
Among the predictions with more than 100 points, the beneﬁt of incorporating the
XC scores is less observable.
Note that the top class score alone is already performing very well for these predictions with more points.
This might be why it is more difﬁcult to get additional beneﬁt from the XC scores on these predictions.
[Table]
Conclusion and future work
To use the explanations quantitatively, we proposed four XC scores to measure the concentration of the attribution values generated for individual predictions.
Applying the four XC scores on the task of classifying TP vs. FP predictions led to over 100% improvement in AUPR on the pedestrian class on both the KITTI and the Waymo datasets compared to simple heuristics such as distance to sensor and number of LiDAR points in bounding box.
Although the XC scores alone could not outperform class score in the TP vs. FP classiﬁcation task, combining class score with the XC scores using an MLP led to notable improvement compared to using class score alone on the pedestrian predictions.
Thus, it is worthwhile to explore the XC scores further for more use cases such as using it in loss functions to improve model performance, or use it as a tool for adversarial or out-of-distribution sample detection.
We also discovered that the XC metrics derived from backprop attributions often outperform those derived from IG attributions on TP vs. FP classiﬁcation for the pedestrian and cyclist objects.
This indicates that explanations that are more understandable to a human observer, such as IG, may not offer superior value when analyzed quantitatively and researchers should not discard simpler explanation methods when exploring quantitative use cases for XAI.
We would like to thank Huawei Noah’s Ark Lab, Canada, for sponsoring our work. We also thank our colleagues Sean Sedwards, Rick Salay, Chengjie Huang, Harry Nguyen and Matthew Pitropov for their support and feedback