(De-)Randomized Smoothing for Decision Stump Ensembles
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Tree-based models are used in many high-stakes application domains such asﬁnance and medicine, where robustness and interpretability are of utmost importance.
Yet, methods for improving and certifying their robustness are severelyunder-explored, in contrast to those focusing on neural networks.
Targeting thisimportant challenge, we propose deterministic smoothing for decision stump ensembles.
Whereas most prior work on randomized smoothing focuses on evaluatingarbitrary base models approximately under input randomization, the key insight ofour work is that decision stump ensembles enable exact yet efﬁcient evaluation viadynamic programming.
Importantly, we obtain deterministic robustness certiﬁcates,even jointly over numerical and categorical features, a setting ubiquitous in the realworld.
Further, we derive an MLE-optimal training method for smoothed decisionstumps under randomization and propose two boosting approaches to improvetheir provable robustness.
An extensive experimental evaluation shows that ourapproach yields signiﬁcantly higher certiﬁed accuracies than the state-of-the-art fortree-based models.
We release all code and trained models at ANONYMIZED.
Introduction
Tree-based models have long been a favourite for making decisions in high-stakes domains such asmedicine and ﬁnance, due to their interpretability and exceptional performance on structured data[1].
However, recent results have highlighted that tree-based models are, similarly to other machinelearning models [2, 3], also highly susceptible to adversarial examples [4–6], raising concerns abouttheir use in high-stakes domains where errors can have dire consequences.
While the robustness of neural models has received considerable attention [7–21], the challengeof obtaining robustness guarantees for ensembles of tree-based models has only been investigatedrecently [4, 22, 23].
However, these initial works only consider numerical features and are based onworst-case approximations, which do not scale well to the difﬁcult (cid:96) p -norm setting.
This Work In this work, we address this challenge and present DRS, a novel D e R andomized
S moothing approach, for constructing robust tree-based models with deterministic (cid:96) p -norm guaranteeswhile supporting both categorical and numerical variables.
Unlike prior work, our method is basedon Randomized Smoothing (RS) [24], an approach that obtains robustness guarantees by evaluating ageneral base model under an input randomization φ ( x ) .
However, in contrast to standard applicationsof RS, which use costly and imprecise approximations via sampling and only obtain probabilisticcertiﬁcates, we leverage the structure of decision stump ensembles to compute their exact outputdistributions for a given input randomization scheme and thus obtain deterministic certiﬁcates.
Our key insight is that this distribution can be efﬁciently computed by aggregating independentdistributions associated with the individual features used by the ensemble.
[Figure]
( x
[Figure] [Figure]
[Figure] [Figure]
[Figure]
.ρ
− 1 (0)
NaturalPrediction successprobability p
y classclass
[Figure]
We illustrate this idea in Fig.
In (a), we show an ensemble of decision stumps over three features( x 1 , x 2 , x 3 ), aggregated to piecewise constant functions over one feature each (discussed in Section 3)and evaluated under the input randomization φ ( x ) , here a Gaussian.
We can compute the independentprobability density functions of their outputs (PDFs) (shown in (b)) directly, by evaluating the(Gaussian) cumulative density function (CDF) over the constant regions.
Aggregating the individual
PDFs (discussed in Section 3), we can efﬁciently compute the exact PDF (c) and CDF (d) of theensemble’s output.
To evaluate and certify the smoothed model, we can now simply look up themedian prediction and success probability, respectively, in the CDF, without requiring sampling.
DRS combines (cid:96) p -norm certiﬁcates over numerical features, computed as described above, withan efﬁcient worst-case analysis for (cid:96) 0 -perturbations of categorical features in order to, for the ﬁrsttime, provide joint certiﬁcates.
To train models amenable to certiﬁcation with DRS, we propose arobust MLE optimality criterion for training individual stumps and two boosting schemes targetingthe certiﬁed robustness of the whole ensemble.
We show empirically that DRS signiﬁcantly improveson the state-of-the-art, increasing certiﬁed accuracies on established benchmarks up to 4-fold .
Main Contributions
Our key contributions are:
• DRS, a novel and efﬁcient Derandomized Smoothing approach for robustness certiﬁcation,enabling joint deterministic certiﬁcates over numerical and categorical variables (Section 3).
• A novel MLE optimality criterion for training decision stumps robust under input random-
ization and two boosting approaches for certiﬁably robust stump ensembles (Section 4).
• An extensive empirical evaluation, demonstrating the effectiveness of our approach and
establishing a new state-of-the-art in a wide range of settings (Section 5)
Background on Randomized Smoothing
For a given base model F : R d → [ C ] , classifying inputs to one of C ∈ Z ≥ 2 classes, Randomized
Smoothing (RS) is a method to construct a classiﬁer G : R d → [ C ] with robustness guarantees.
For arandomization scheme φ : R d → R d , we deﬁne the success probability p
y :=
P x (cid:48) ∼ φ ( x ) [ F ( x (cid:48) )
=
y ] and
G ( x ):=arg max c ∈
[ C ] p y .
Depending on the choice of φ , we obtain different certiﬁcates of the form:
Theorem 2.1 (Adapted from Cohen et al.
[24], Yang et al.
If P ( F ( φ ( x )) = y ) :
= p y ≥ p yand p y > 0 .
5 , then G ( x + δ ) =
y for all δ satisfying (cid:107) δ (cid:107) p < R with R :=
ρ ( p y ) .
[Table]
R :=
ρ ( p
[Table]
) (cid:96)x + Unif
[Equation]
In particular, we present two instantiations that we utilize throughout this paper in Table 1, where Φ − 1 is theinverse Gaussian CDF.
Similar results, yielding other(cid:96) p -norm certiﬁcates, can be derived for a wide rangeof input randomization schemes [25, 26].
Note that, byusing more information than just p y , e.g., p c for therunner-up class c , tighter certiﬁcates can be obtained[24, 27].
Once p y is computed, we can directly calculate the certiﬁable radius R :=
ρ ( p y ) .
For abroader overview of variants of Randomized Smoothing, please refer to Section 6.
For most choices of f and φ , the exact success probability p y can not be computed efﬁciently.
Thus a lower bound p y is estimated with conﬁdence 1 − α (typically α = 10 − 3 ) using Monte Carlosampling and the Neyman-Pearson lemma [28].
Not only is this extremely computationally expensive,as typically 100 000 samples have to be evaluated per data point, but this also severely limits themaximum certiﬁable radius (see Fig. 6) and only yields probabilistic guarantees.
Additionally, if thenumber of samples is not sufﬁcient for the statistical test, the procedure will abstain from classifying.
In the following, we will show how considering a speciﬁc class of models f allows us to computethe success probability p
y exactly, overcoming these drawbacks, and thus invoke ρ ( p y ) to computedeterministic certiﬁcates over larger radii, orders of magnitude faster than RS.
Derandomized Smoothing for Decision Stump Ensembles
Tree-based models such as decision stump ensembles often combine exceptional performance on tabular data [1] with good interpretability, making them ideal for many real-world high-stakes applications.
Here, we propose DRS, a method to equip them with deterministic robustness guarantees.
In particular, we show that their structure permits an exact evaluation under isotropic input randomizationschemes, such as those discussed in Section 2, and thus deterministic smoothing.
Stump
Ensembles We deﬁne a decision stump as f m ( x ) =
γ l,m + ( γ r,m − γ l,m ) 1 x j m >v m , withleaf predictions γ l,m , γ r,m ∈ [0 , 1] , split position v m , and split variable j m .
We construct unweightedensembles, particularly suitable for Smoothing [29], of M such stumps ¯ f M : R d (cid:55)→ [0 , 1] as
[Equation]
and treat them as a binary classiﬁers 1 ¯ f M ( x ) > 0 .
While our approach is extensible to multi-classclassiﬁcation by replacing the scalar leaf predictions γ with prediction-vectors, assigning a score perclass, we focus on the binary case in this work.
Stump Ensemble We now deﬁne a smoothed stump ensemble ¯ g M along the lines of
Randomized Smoothing as discussed in Section 2, by evaluating ¯ f M not only on the original input
xbut rather on a whole distribution of x (cid:48) ∼ φ ( x ) :
[Equation]
In this work, we consider randomization schemes φ ( x ) that are (i) isotropic, i.e., the dimensions ofx (cid:48) ∼ φ ( x ) are independently distributed, and (ii) permit an efﬁcient computation of their marginalcumulative distribution functions (CDF).
This includes a wide range of distributions, e.g., the Gaussianand Uniform distributions used in Table 1 and others commonly used for RS [25].
[Equation]
ppp
By denoting the model CDF as ¯ F M, x
[Equation]
≤
[Equation]
[Equation]
we can alternatively deﬁne ¯ g M as ¯ g M ( x ) :
= 1 − ¯ F M, x (0 . 5) , 102 which will become useful later.
For a label y ∈ { 0 , 1 } we obtain 103 the success probability p y =
| y − ¯ F M, x (0 . 5) | of predicting y for 104 a sample from φ ( x ) .
Meta-Stumps To evaluate p y exactly as illustrated in Fig.
1, wegroup the stumps constituting an ensemble by their split variablej m to obtain one meta-stump ˜ f
i per feature i .
The key idea is thatoutputs of these meta-stumps are now independently distributedunder isotropic input randomization (illustrated in Fig.
1 (b)),allowing us to aggregate them efﬁciently later on.
We showcase this in Fig.
2, where two stumps ( f 1 and f 2 ) arecombined into the meta-stump ˜ f i .
Formally, we have
[Equation]
deﬁne M i = |I
i | and rewrite our ensemble as ¯ f M ( x ) :
= 1 M (cid:80) di =1 ˜ f
i ( x i ) .
Every meta-stump 114 can be represented by its split positions v i,j , sorted such that v i,j ≤ v i,j +1 , and its predictionsγ i,j = (cid:80) j − 1 t =
γ r,t + (cid:80) |I
i | t =
j γ l,m on each of the resulting |I
i | + 1 regions, written as ( γ , v ) i .
function C OMPUTE PDF( { ( Γ , v ) i } di =1 , x , φ ) pdf
[ i ][ t ] = 0 for t ∈ [ M · ∆ + 1] , i ∈ [ d ] pdf [0][0] =
(cid:46)
For 0 stumps all probability mass is onfor i
= 1 to d do for j = 1 to M i do for t = 0 to M · ∆ + 1 − Γ i,j do pdf
[ i ][ t + Γ
i,j ] = pdf
[ i ][ t + Γ
i,j ] + pdf [ i − 1][ t ] · P x (cid:48)
i ∼ φ ( x ) [ v i,j − 1 ≤ x (cid:48)
i ≤ v i,j ] return pdf
CDF Computation
Now we leverage the independence of our meta-stumps’ output distributionsunder an isotropic input randomization scheme φ to compute the PDF of their ensemble efﬁcientlyvia dynamic programming (DP) (illustrated in Fig.
1 (c) and explained below).
Given its PDF, we cantrivially compute the ensemble’s CDF ¯ F M, x , allowing us to evaluate the smoothed model exactly(illustrated in Fig.
1 (d)).
This efﬁcient CDF computation constitutes the core of DRS.
In more detail, we observe that the PDF of a stump ensemble is the convex sum of exponentiallymany ( O ((max i I i ) d ) )
Dirac delta distributions.
To avoid this exponential blow-up, we discretizeall leaf predictions γ to a grid of ∆ values (typically ∆ = 100 ), when constructing the smoothedmodel ¯ g M .
For each γ i,j , we deﬁne a corresponding Γ i,j ∈ { 0 , . . .
, M i · ∆ } such that γ i,j = Γ i,j ∆ .
125 Now, we construct a DP-table, where every entry pdf[i][t] corresponds to the weight of the Diracdelta at position t ∆ M after considering the ﬁrst i meta-stumps (in any ﬁxed but arbitrary order).
We outline the PDF computation in Algorithm [-1 and provide an intuition below. We initializepdf[0] [*] by allocating all probability mass to t = 
 ( pdf[0][0]=1 ).-]1.
After initially allocating all probability mass tot = 0 , we account for the effect of the i th meta-stump, by updating the i th row in the DP-table withpdf[i][t] = (cid:80) j p j pdf[i-1][t − Γ i,j ] where p j is the probability P x (cid:48)
i ∼ φ ( x ) [ v i,j − 1 ≤ x (cid:48)
i ≤ v i,j ]of the randomized x (cid:48)
i lying between v i,j − 1 and v i,j (padded with −∞ and ∞ on the left and right, 131 respectively). This is illustrated in Fig.
After termination, the last line of the DP-table pdf[d]contains the full PDF (see Fig. 1(c)).
Formally we summarize this in the theorem below, delaying aformal proof to App.
A.1:
Theorem 3.1.
For z ∈ [0 , 1] , ¯ F M, x ( z ) =
(cid:80) (cid:98) zM ∆ (cid:99) t = pdf
[ d ][ t ] describes the exact CDF and thussuccess probability p y
= P x (cid:48) ∼ φ ( x ) [ ¯ f M ( x (cid:48) )
=
y ]
= | y − ¯ F M, x (0 . 5) | for y ∈ { 0 , 1 } .
Note that the presented algorithm is slightly simpliﬁed, and we actually only have to track the rangeof non-zero entries of one row of the DP-table.
This allows us to compute the full PDF and thuscertiﬁcates for smoothed stump ensembles very efﬁciently, e.g., taking only around 1 .
2 s total for the
MNIST 2 VS .
6 task (around 2 .
000 data points and over 500 stumps).
Certiﬁcation Recall from Section 2 that, given the success probability p y , robustness certiﬁcationfor (cid:96) p -norm bounded perturbations reduces to computing the maximal certiﬁable robustness radius
R = ρ ( p y ) .
For all popular (cid:96) p -norms, ρ (and its inverse ρ − 1 ; used shortly) can be either evaluatedsymbolically [24, 25] or precomputed efﬁciently [30, 31], such that the core challenge of certiﬁcationbecomes computing (a lower bound to)
p y , which we solve efﬁciently via Theorem 3.1.
Alternatively,for a given target radius r , we need to check whether p y ≥ ρ − 1 ( r ) by equivalently calculating
[Equation]
and checking ¯ g M,r ( x ) > 0 .
This corresponds to asserting that class y is predicted at least z percentof the time.
Here, the inverse CDF ¯ F − 1 M, x ( z ) can be efﬁciently evaluated using the step-wise ¯ F M, xcomputed via Theorem 3.1.
We will see in Section 4 that this view is useful when training stumpensembles for certiﬁability.
Finally, we want to highlight that this approach can be used with allcommon randomization schemes yielding certiﬁcates for different (cid:96) p -norm bounded adversaries.
Categorical Variables & Joint Certiﬁcates For practical applications, it is essential to handle bothnumerical and categorical features jointly.
To consider a categorical feature x i ∈ { 1 , . .
, d i } in ourstump ensemble, we construct a d i -ary stump ˜ f
i : [ d i ] → [0 , 1] returning a value γ i,j correspondingto each of the d i categorical values and treated as a meta-stump with M i = 1 for normalization.
To provide certiﬁcates in this setting, we propose a novel scheme combining an arbitrary (cid:96) p -normcertiﬁcate of radius r p over all numerical features, computed as discussed above, with an (cid:96) 0 certiﬁcateof radius r 0 over all categorical features C , computed using an approach adapted from Wang et al.
Conceptually, we compute the worst case effect of every individual categorical variable independently,greedily aggregate these worst case effects, and account for them in our ensemble’s CDF.
Given a meta-stump’s prediction on a concrete sample q i = ˜
f i ( x i ) as well as its maximal andminimal output u i and l i , respectively, we compute the maximum and minimum perturbation effectto δ i
=
u
i − q i M and δ i
=
l i − q i M , respectively.
Given the set of categorical features C , we can computethe worst-case effect when perturbing at most r 0 samples as
[Figure]
by greedily picking the r 0 largest δ i .
For δ r 0 we proceed analogously.
Shifting the CDF, computed as above, by δ and δ forsamples with labels y
=
and y
= 1 , respectively, before computing the success probability p y , allows us to account for theworst-case categorical perturbations exactly.
We illustrate thisfor a sample with y = 0 in Fig.
3, where we show the CDFsobtained by all possible perturbations of at most r 0 categoricalvariables, bounded to the right by those obtained by shifting theoriginal by δ r 0 .
Note that here no smoothing over the categoricalvariables is done or required, making inference trivial.
Training for and with Derandomized Smoothing
To obtain large certiﬁed radii via smoothing, the base model has to be robust to the chosen randomization scheme.
To train robust decision stump ensembles, we propose a robust MLE optimality criterionfor individual stumps (Section 4.1) and two boosting schemes for whole ensembles (Section 4.2).
4.1 Independently MLE-Optimal Stumps
To train an individual stump f m ( x )
=
γ l,m + ( γ r,m − γ l,m ) 1 x jm >v m , its split feature j m , split 180 position v m , and leaf predictions γ l,m , γ r,m have to be determined.
We choose them in an MLEoptimal fashion with respect to the randomization scheme φ , starting with v m , as follows:
We considerthe probabilities p l,i =
P x (cid:48) ∼ φ ( x i )
[ x (cid:48)
j m ≤ v m ] and p r,i = 1 − p l,i of x (cid:48)
i lying to the left or the right 183 of v m , respectively, under the input randomization scheme φ .
For an i.i.d.
dataset with n samples( x i , y i ) ∼ ( X , Y ) , we deﬁne the probabilities p yj = 1 n (cid:80) { i | y i =
y } p j,i of picking the j ∈ { l, r } leaf,conditioned on the target label, and p j = p
j + p 1 j as their sum to compute the entropy impurity H entropy [32] as
[Equation]
We then choose the v m approximately minimizing H entropy via line-search.
After ﬁxing v m this way,we compute the MLE-optimal leaf predictions γ φ, MLE l and γ φ, MLE r as:
[Equation]
r = arg max γ l ,γ r
P [ Y | φ ( X ) , f m ]
= arg max γ l ,γ r m (cid:88) i =
E x (cid:48) ∼ φ ( x i )
[log P [ y i | x (cid:48) , f m ]]
= arg max γ l ,γ r m (cid:88) i ∈{ i | y i =0 } p l,i log(1 − γ l ) + p r,i log(1 − γ r ) + m (cid:88) i ∈
{ i | y i =1 } p l,i log( γ l ) + p r,i log( γ r ) =
arg max γ l ,γ r p 0 l log(
[Equation] l ) + p 0 r log( [Equation] γ r ) + p 1 l log( γ l ) + p 1 r log( γ r ) , where the second line is obtained by splitting the sum over samples by class and explicitly computingthe expectation.
We solve the maximization problem by setting the ﬁrst derivatives ∂∂γ l and ∂∂γ r ofour optimization objective to zero and checking its Hessian to conﬁrm that
[Equation] [Equation]
are indeed maxima.
We show in App.
A.2 that γ φ, MLE l , γ φ, MLE l , and v m are even jointly MLE-optimal,when v m is chosen as the exact instead of an approximate minimizer of the entropy impurity.
Ensembling To train an ensemble of independently MLE-optimal decision stumps, we sequentiallytrain one stump for every feature j m ∈
[ d ] and construct an ensemble with equal weights, rejectingstumps with an entropy impurity H entropy above a predetermined threshold.
4.2 Boosting Stump Ensembles for Certiﬁable Robustness
[Figure]
Decision stumps trained this way maximize the expected likelihood under the chosen randomization scheme.
Assuming (due tothe law of large numbers) a Gaussian output distribution, this corresponds to optimizing for the median output, which determinesthe clean prediction.
However, certiﬁed correctness at a given radius r is determined by the prediction y (cid:48)
( x , r ) =
¯ F − 1 m − 1 , x ( z ( r ))at the z ( r ) :=
| y − ρ − 1 ( r ) | percentile of the output distribution.
Where we call y (cid:48) the certiﬁable prediction , as certiﬁcation is nowequivalent to checking y = 1 y (cid:48)
( x ,r ) > 0 .
5 (Eq.
This difference 207 is illustrated in Fig.
4, where the clean prediction is correct (class 1) while the certiﬁable predictionis incorrect.
To align our training objective better with certiﬁed accuracy, we propose two novelboosting schemes along the lines of the popular T REE B OOST [33] and A DA B OOST [34].
Gradient Boosting for Certiﬁable Robustness
The key idea of gradient boosting is to compute thegradient of a loss function with respect to an ensemble’s outputs and then add a model to the ensemble,making a prediction along this gradient direction.
Implementing this idea, we adapt T REE B OOST [33]to propose R OB T REE B OOST :
At a high level, we add stumps to the ensemble, which aim to predictthe residual between the target label and the current certiﬁable prediction.
Concretely, to add the m thstump to our ensemble, we begin by computing the current ensemble’s certiﬁable predictions y (cid:48)
( r ) ata target radius r and then deﬁning the pseudo labels ˜
y
=
y − y (cid:48)
( r ) as the residual between the targetlabels y and the certiﬁable predictions y (cid:48) ( r ) .
This yields a regression problem, which we tackle bychoosing a feature j m and split threshold v m (approximately) minimizing MSE impurity under inputrandomization before computing γ l,m and γ r,m as approximate minimizers of the cross-entropy lossover the whole ensemble.
Please see App.
A.3 for a more detailed discussion of R OB T REE B OOST .
Adaptive Boosting for Certiﬁable Robustness
The key idea of adaptive boosting is to build anensemble by iteratively training models, weighted based on their error rate, while adapting sampleweights based on whether they are classiﬁed correctly.
We build on A DA B OOST [34] to propose
R OB A DA B OOST :
We construct an ensemble of K stump ensembles via hard voting, where everyensemble is weighted based on its certiﬁable accuracy.
To train a new ensemble, we increase theweights of all samples that are currently not classiﬁed certiﬁably correct at a given radius r .
Wechoose stump ensembles instead of individual stumps as base classiﬁers because single stumps cannot reach the success probabilities under input randomization required for certiﬁcation.
To computethe certiﬁable radius for such an ensemble ¯ F K , we compute the certiﬁable radii R k of the individualstump ensembles ¯ f kM , sort them in decreasing order such that R k ≥ R k +1 and obtain the largestradius R k such that the weights of the ﬁrst k ensembles sum up to more than half of the total weights.
Please see App.
A.4 for a more detailed discussion of R OB A DA B OOST .
Experimental Evaluation
In this section, we empirically demonstrate the effectiveness of DRS in a wide range of settings.
Weshow that DRS signiﬁcantly outperforms the current state-of-the-art for certifying tree-based modelson established benchmarks, using only numerical features (Section 5.1), before highlighting its novelability to obtain joint certiﬁcates on a set of new benchmarks (Section 5.2).
Finally, we perform anablation study, investigating the effect of DRS’s key components (Section 5.3).
[Table]
Experimental Setup We implement our approach in PyTorch [35] and evaluate it on Intel Xeon
Gold 6242 CPUs and an NVIDIA RTX 2080Ti.
We compare to prior work on the D IABETES[36], B REAST C ANCER [37], FMNIST-S HOES [38], MNIST 1 VS .
5 [39], and MNIST 2 VS .6 [39] datasets and are the ﬁrst to provide joint certiﬁcates of categorical and numerical features,demonstrated on the A DULT [37] and C REDIT [37] datasets.
For a more detailed description of theexperimental setup, please refer to App.
B.
5.1 Certiﬁcation for Numerical Features
In Table 2, we compare the certiﬁed accuracies obtained via DRS on ensembles of independently
MLE optimal stumps (Independent) or boosted stump ensembles (Boosting) to the current state-ofthe-art, Wang et al.
[23], using established benchmarks [23].
Independently MLE Optimal Stumps We ﬁrst consider stump ensembles trained without boostingas described in Section 4.1 and observe that DRS obtains higher certiﬁed accuracies in all settingsand higher natural accuracies in most.
For example, on MNIST 2 VS . 6 , we increase the certiﬁedaccuracy at an (cid:96) 2 radius of r 2 = 0 .
8 from 23 .
0% to 89 .
7% , almost quadrupling it compared to Wanget al.
[23], while also improving natural accuracy from 86 .
3% to 96 .
Boosting for Certiﬁed Accuracy Leveraging the boosting techniques introduced in Section 4.2,
R OB T REE B OOST for B REAST C ANCER and D IABETES and R OB A DA B OOST for FMNIST-S HOES ,
MNIST 1 VS .
5 , and MNIST 2 VS .
6 , we increase certiﬁable and natural accuracies even further inmost settings.
For example, compared to our independently trained stump ensemble, we improvethe certiﬁed accuracy for MNIST 1 VS .
5 at an (cid:96) 1 -radius of r 1 = 1 .
from 94 .
2% to 98 .
1% and for
B REAST C ANCER at an (cid:96) 2 -radius of r 2 = 0 .
7 from 75 .
2% to 82 .
5.2 Joint Certiﬁcates for Categorical and Numerical Features
[Figure]
In Table 3, we compare models using only numerical, only categorical, or both types of features with regards to their balancedcertiﬁed accuracy (BCA) (accounting for class frequency) atdifferent combinations of (cid:96) 2 - and (cid:96) 0 -radii for numerical andcategorical features, respectively.
We observe that models using both categorical and numerical features perform notablybetter on clean data, highlighting the importance of utilizingand thus also certifying them in combination.
Moreover, categorical features make the model signiﬁcantly more robust to(cid:96) 2 perturbations, e.g., at (cid:96) 2 -radii > 0 .
75 , they improve certiﬁedaccuracies, even when 2 categorical features (of only 8 and7 for A DULT and C REDIT , respectively) are adversarially perturbed.
We visualize this in Fig.
5,showing BCA over (cid:96) 2 -perturbation radius and conﬁrm that the model utilizing only numerical features(dotted line) loses accuracy much quicker with perturbation magnitude than the model leveragingcategorical variables (solid lines).
As we are the ﬁrst to tackle this setting, we do not compare toother methods but provide more detailed experiments in App.
C.1.
[Table]
5.3 Ablation Study
We ﬁrst illustrate the effectiveness of our derandomization approach, before demonstrating the beneﬁtof training with our MLE optimality criterion and investigating the effect of the noise level on DRS.
[Figure]
Derandomized vs Randomized Smoothing In Fig.
6, wecompare DRS, (dotted line) and sampling-based RS (solidlines), with respect to certiﬁed accuracy over (cid:96) 2 radii.
Weobserve that the sampling-based estimation of the successprobability in RS signiﬁcantly limits the obtained certiﬁable radii.
While this effect is particularly pronounced forsmall sample counts n , increasing the maximum certiﬁableradius, visible as the sudden drop in certiﬁable accuracy,requires an exponentially increasing number of samples,making the certiﬁcation of large radii intractable.
DRS, incontrast, can compute exact success probabilities and thusdeterministic guarantees for much larger radii, yielding a33 .
1% increase in ACR compared to using n = 100 000samples.
Additionally, DRS is multiple orders of magnitude faster than RS, here, only requiringapproximately 6 .
45 · 10 − 4 s per sample.
For more extensive experiments, please refer to App.
C.2.
[Figure]
MLE Optimality Criterion In Table 4, we evaluate ourrobust MLE optimality criterion (MLE) by comparing itto the standard entropy criterion applied to samples drawnfrom the input randomization scheme (Sampling) or theclean data (Default).
We observe that the ensemble trainedon the clean data (Default) suffers from a mode collapsewhen evaluated under noise.
In contrast, both approachesconsidering the input randomization perform much better,with our robust MLE approach outperforming samplingby a signiﬁcant margin, especially at large radii.
For moreextensive experiments, please refer to App.
C.3.
[Figure]
Effect of Noise Level In Fig.
7, we compare the certiﬁedaccuracy over (cid:96) 1 -radii for a range of different noise magnitudes λ and ensembles of independently MLE optimalstumps.
We observe that at large perturbation magnitudes,we obtain stumps that ‘think outside the (hyper-)box’, i.e.,choose splits outside of the original data range, makingtheir ensembles exceptionally robust, even at large radii.
In particular, we obtain a certiﬁable accuracy of 87 .
3% atradius r 1 = 4 . 0 , while the state-of-the-art achieves only83 .
8% at r 1 = 1 .
We provide more experiments forvarying noise magnitudes in App.
C.4.
Related Work
(De-)Randomized Smoothing Probabilistic certiﬁcation methods [40, 41, 24] are a popular approachfor obtaining robustness certiﬁcates for a wide range of tasks [31, 42–45] and threat models [25, 26,46, 47, 27, 30, 31, 42, 48–52].
These methods follow the general blueprint discussed in Section 2and consider arbitrary base classiﬁers, though specially trained [53–55], and can thus, in contrastto our work, not leverage their structure.
Speciﬁcally designed randomization schemes [50, 52]enable efﬁcient enumeration and thus a deterministic certiﬁcate for, e.g., patch attacks or (cid:96) 1 -normperturbations.
In contrast to these approaches, we permit arbitrary isotropic continuous randomizationschemes, allowing us to leverage comprehensive results on RS to obtain robustness guarantees againsta wide range of (cid:96) p -norm bounded adversaries [25].
Certiﬁcation and Training of Tree-Based Models In the setting of (cid:96) ∞ robustness, where everyfeature can be perturbed independently, various methods have been proposed to train [4, 22, 56, 57]and certify [22, 58–60] robust decision trees and stumps.
However, (cid:96) ∞ robust models are stillvulnerable to other (cid:96) p norm perturbations [61, 62], which cover many realistic perturbations betterand are the focus of this work.
There, the admissible perturbation of one feature depends on theperturbations of all others, making the above approaches leveraging their independence not applicable.
On the other hand, Kantchelian et al.
[63] discuss complete robustness certiﬁcation of tree ensemblesin the (cid:96) p -norm setting via MILP.
However, this approach is intractable in most settings due its CoNP-complete complexity.
Wang et al.
[23] propose an efﬁcient but incomplete DP-based certiﬁcationalgorithm for stump ensembles based on over-approximating the maximum perturbation effect inthe (cid:96) p -norm setting.
While similarly fast as our approach, we show empirically in Section 5 that
DRS obtains signiﬁcantly stronger certiﬁcates.
Wang et al.
[23] further introduce an incompletecertiﬁcation algorithm for tree ensembles, which is based on computing the distance between thepre-image of all trees’ leaves and the original sample.
As they report signiﬁcantly worse results usingthis approach than with stump ensembles, we omit a detailed comparison.
Limitations and Societal Impact
Limitations While able to handle arbitrary stump ensembles, and being extensible to arbitrarydecision trees, DRS can not handle arbitrary ensembles of decision trees.
However, as these havebeen shown to be signiﬁcantly more sensitive to (cid:96) p -norm perturbations than stump ensembles [23], webelieve this limitation to be of little practical relevance.
Further, like all Smoothing-based approaches,we construct a smoothed model from a base classiﬁer and only obtain robustness guarantees for theformer.
In contrast to standard Randomized Smoothing approaches, we can, however, evaluate thesmoothed model exactly and efﬁciently.
Societal Impact
As our contributions improve certiﬁed accuracy and certiﬁcation radii while retaininghigh natural accuracy, they could help make real-world AI systems more robust and thus generallyamplify both any positive or negative societal effects.
Further, while we achieve state-of-the-artresults, these may not be sufﬁcient to guarantee robustness in real-world deployment and could givepractitioners a false sense of security, leading to them relying more on our models than is justiﬁed.
Conclusion
We propose DRS, a (De-)Randomized Smoothing approach to robustness certiﬁcation, enabling jointdeterministic certiﬁcates over numerical and categorical variables for decision stump ensembles byleveraging their structure to compute their exact output distributions for a given input randomizationscheme.
The key insight enabling this is that this output distribution can be efﬁciently computed byaggregating independent distributions associated with the individual features used by the ensemble.
We additionally propose a robust MLE optimality criterion for training individual decision stumpsand two boosting schemes improving an ensemble’s certiﬁable accuracy.
Empirically, we demonstratethat DRS signiﬁcantly outperforms the state-of-the-art for tree-based models in a wide range ofsettings, obtaining up to 4-fold improvements in certiﬁable accuracy.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] We substantiate all claims theoretically (Section 3 and Section 4) or empirically (Section 5), as appropriate.
(b) Did you describe the limitations of your work?
[Yes] We describe the limitations in
Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss
societal impacts in Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and Section 4 and App. A.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3 and Section 4 and App. A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] We will include them as supplemental material and will release them to the public upon publication.
All datasets we use are publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We provide full training details in App.
B.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[N/A]
Both our training and certiﬁcation methods are
deterministic (up to ﬂoating point errors), thus we do not report error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We provide resource type details in
Section 5 and timing details in App.
B.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]