T OWARDS U NDERSTANDING THE R OBUSTNESS A GAINST E VASION A TTACK ON C ATEGORICAL I NPUTS
A BSTRACT
Characterizing and assessing the adversarial risk of a classiﬁer with categorical inputs has been a practically important yet rarely explored research problem.
Conventional wisdom attributes the difﬁculty of solving the problem to its combinatorial nature.
Previous research efforts tackling this problem are speciﬁc to use cases and heavily depend on domain knowledge.
Such limitations prevent their general applicability in real-world applications with categorical data.
Our study novelly shows that provably optimal adversarial robustness assessment is computationally feasible for any classiﬁer with a mild smoothness constraint.
We theoretically analyze the impact factors of adversarial vulnerability of a classiﬁer with categorical inputs via an information-theoretic adversarial risk analysis.
Corroborating these theoretical ﬁndings with a substantial experimental study over various real-world categorical datasets, we can empirically assess the impact of the key adversarial risk factors over a targeted learning system with categorical inputs.
I NTRODUCTION
Categorical data pervasively exist in real-world safety-critical Machine-Learning-as-a-Service (MLaaS) applications, such as ML-driven intrusion detection and digital healthcare.
The vulnerability to attacks by intentionally crafting categorical signatures raises concerns on trust and utility of the MLbased analytic services.
Characterizing and assessing adversarial robustness on categorical data can thus help evaluate the reliability of the core ML models and ﬂag potential evading efforts.
For a classiﬁer 𝑓 with categorical inputs 𝑥 , the adversarial risk of 𝑓 under evasion attack can be formulated as follows.
Deﬁnition 1 𝑓 : 𝑥 → 𝑦 denotes a classiﬁer with categorical inputs 𝑥 .
Let 𝜇 𝑥,𝑦 denote the joint distribution of ( 𝑥,𝑦 ) .
The expected adversarial risk of the classiﬁer 𝑓 under evasion attack is formulated as:
[Equation]
(1) where ℓ is the misclassiﬁcation loss function, 𝑥 and ˆ 𝑥 are an unperturbed and the generated adversarial sample respectively.
𝜀 denotes the attack budget of evasion attack.
𝑥 is correctly classiﬁed ( 𝑓 ( 𝑥 )
= 𝑦 ).
Intuitively, a classiﬁer 𝑓 is more robust against adversarial perturbations, if its adversarial risk is low given the attack budget limiting the number of changed categorical features.
Unlike continuous data, a categorical variable can be valued with only one categorical value among others.
These categorical values have no intrinsic ordering to the categories.
Evasion attack manipulating categorical inputs is in nature an NP-hard knapsack problem .
The discontinuous nature raises two fundamental yet rarely addressed questions to evaluate the adversarial risk on categorical data in practice:
[-• Q
-]Q1 What are the key factors determining 𝑓 ’s adversarial risk R 𝑎𝑑𝑣𝜀 on categorical data ?
Q
For a general classiﬁer 𝑓 , can we assess the adversarial risk of 𝑓 with categorical inputs with provably accuracy guarantee ?
Despite recent efforts of adversarial vulnerability exploration with discrete data, both questions remain open for several reasons.
First, the discontinuity of categorical space prevents the direct use of the previous progress on adversarial risk analysis with continuous data.
The local subspace assumption of 𝑙 𝑝 -bounded adversarial attacks does not apply to the categorical features (Hein & Andriushchenko, 2017;Wangetal.
,2018;Fawzietal. ,2016;Gilmeretal.
,2018;D.Yinetal. ,2019;Khim&Loh,2018;Tu etal.
,2019).Second,mostpracticesofdiscreteadversarialattackaredomainspeciﬁcanddependheavilyondomainknowledge.(Bojchevski&G¨unnemann,2019;Bojchevski&G¨unnemann,2019;Zugner &Gunnemann,2019)focusonbuildingdifferentiablesurrogatefunctionstoGraphNeuralNetworksto facilitatesearchingforfeasiblepoisoningeditsovergraphstructuresandnodeattributes.(Narodytska &Kasiviswanathan,2017;Croce&Hein,2019)conduct 𝐿 0 -norm perturbations only within local image areas containing sensitive features for image classiﬁcation.
(Qi et al., 2019; Wang et al., 2020) require non-negativity on the parameters of deep neural networks to deliver provably accurate greedy attacks via submodular function optimization.
The non-negativity constraint is unnatural for real-world
ML practices.
It does harm the utility of the classiﬁer.
For a more general classiﬁer with categorical inputs, a provably optimal and domain-agnostic method for attack and adversarial risk evaluation is yet to establish.
Using greedy search or the well-known Branch-and-Bound method on a general knapsack problem provides no optimality guarantee of the solutions, thus can produce arbitrarily bad results.
Our study aims to address these two questions mentioned above from both theoretical and empirical perspectives.
First , we derive an information-theoretic characterization of the adversarial risk of a classiﬁer.
It unveils that the informativeness of the input categorical instance , the sensitivity of the perturbed categorical features and the information geometry property of the targeted classiﬁer are the three key factors jointly determining adversarial vulnerability of the classiﬁer.
Second , our study adopts an assess-by-attack strategy.
We show that assessing adversarial robustness of any measurable classiﬁer with categorical inputs can be cast to a weakly submodular maximization problem, with a mild smoothness condition.
It can thus be solved using a simple yet efﬁcient greedy attack strategy with provable approximation guarantees.
The theoretical ﬁndings not only explain the empiricalsuccessofthegreedysearchstrategytogenerateadversarialtextualandimagesamples(Gong etal.
,2018;Yangetal.
,2018;Narodytska&Kasiviswanathan,2017),butalsopavethewaytoadomainagnosticadversarialrobustnessassessmentoncategoricaldatawithprovableoptimalityguarantees .
Third , we instantiate the domain-agnostic adversarial risk characterization and assessment with a widely used DNN classiﬁer, i.e. Long Short-Term Memory (LSTM) and three different categorical datasets in Section.4.
These datasets are collected from various real-world applications.
The experimental results conﬁrm the impact of the 3 risk factors over the adversarial vulnerability of the DNN classiﬁer.
R ELATED W ORK
Tremendous efforts have been made to vulnerability measurement of a classiﬁer under evasion attack (Hein & Andriushchenko, 2017;
Wang et al., 2018; Fawzi et al., 2016; Gilmer et al., 2018; Weng et al., 2019; Sinha et al., 2018;
Cohen et al., 2019; Shi et al., 2020; D.Yin et al., 2019; Khim & Loh, 2018;
Tu et al., 2019).
Most of the previous works focus on evaluating robustness against 𝑙 𝑝 -norm perturbations on continuous data.
They all assume adversarial samples locate within a smooth 𝑙 𝑝 -ball around an input instance, which doesn’t hold for categorical data.
In contrast, Tu et al.
(2019) covers both numerical and categorical data.
It bounds the adversarial risk with a local worst-case risk over a 𝑝 -wassernstein ball centered at the training data distribution.
This work associates the adversarial risk of a classiﬁer with its rademacher complexity.
Nevertheless, we argue that the adversarial vulnerability of a classiﬁer is determined by not only the characteristics of the classiﬁer, e.g. model complexity, but also the properties of the training/testing data instances.
Pioneering works of evasion attacks with categorical inputs depend on domain-speciﬁc knowledge to facilitate the attack exploration.
Kuleshov et al.
; Papernot et al.
; Miyato et al.
Samanta & Mehta (2017); Gao et al.
; Yang et al.
; Gong et al.
; Ebrahimi et al.
(2018); Narodytska & Kasiviswanathan (2017); Croce & Hein (2019) focus on replacing individual words/phrases to cheat text classiﬁers, or modifying pixel intensities to bias image classiﬁcation results.
These methods use heuristic semantic rules, e.g., replacing words with manually deﬁned candidate synonyms and constraining the word change to preserve readability and semantic integrity.
Narodytska & Kasiviswanathan (2017); Croce & Hein (2019) narrow down the search range to the pixels with high pixel-wise sensitivity for image classiﬁcation.
Despite of the sounding empirical results, there is no guarantee on a successful attack within the attack budget.
Bojchevski & G¨unnemann (2019); Bojchevski & G¨unnemann (2019); D.Zugner et al.
; D. Z¨ugner & G¨unnemann (2018); Akbarnejad & G¨unnemann (2019) adopt edge-ﬂipping and node attribute perturbation to poison graph data mining pipelines, e.g., graph neural networks and graph embedding models.
The key idea is to introduce relaxed surrogate functions to the combinatorial attack objective and then solve the relaxed optimization problem instead.
Notably, D.Zugner et al.
; D. Z¨ugner & G¨unnemann (2018); Akbarnejad & G¨unnemann (2019) deﬁne the attack objective as a sum of smallest eigenvalues of the adjacency matrix of a given graph.
Though it is not explicitly claimed, it is intrinsically a submodular maximization problem.
Qi et al.
; Wang et al.
(2020) unveil that simple greedy search can deliver provably effective attacks against DNN classiﬁers without domain-speciﬁc heuristics, if all the link weights between neurons are non-negative.
Non-negativity of the parameter guarantees strict submodularity of the attack objective.
However, it brings signiﬁcantly deterioration of the classiﬁer’s accuracy, which is not applicable for real-world learning tasks.
R OBUSTNESS C HARACTERIZATION AND A SSESSMENT
We assume an input instance 𝑥
= { 𝑥 1 ,𝑥 2 ,𝑥 3 ,...,𝑥 𝑛 } of 𝑛 categorical attributes.
Each 𝑥 𝑖 takes any of 𝑚 ( 𝑚 ≥ 1 ) categorical values.
The classiﬁer 𝑓 outputs decision probabilities 𝑓 𝑦 𝑘 ( 𝑘 = 1 , 2 , 3 ,...,𝐾 ) with respect to different class labels.
In practices, each categorical value of 𝑥 𝑗𝑖 is cast to a 𝐷 -dimensional pretrained embedding vector, e.g., 𝑒 𝑗𝑖 ∈ R 𝐷 ,𝑗 = 1 , 2 ,...,𝑚 .
To represent an instance 𝑥 with the embedding vectors of its category values, we deﬁne binary variables
𝑏 = { 𝑏 𝑗𝑖 } , 𝑖 = 1 , 2 ,...,𝑛 , 𝑗 = 1 , 2 ,...,𝑚 , where 𝑏 𝑗𝑖 =when the 𝑗 -th attribute value is present for 𝑥 𝑖 and 𝑏 𝑗𝑖 = 0 otherwise.
An instance 𝑥 can then be represented by an R 𝑛 ∗ 𝑚 ∗ 𝐷 tensor with 𝑥 { 𝑖,𝑗, : } = 𝑏 𝑗𝑖 𝑒 𝑗𝑖 .
Let ˆ 𝑏 = { ˆ 𝑏 𝑗𝑖 } indicate the adversarial modiﬁcations introduced into 𝑏 .
For a perturbed ˆ 𝑥 , its ˆ 𝑏 ≠ 𝑏 .
Depending on the type of attacks to implement, e.g., insertion , deletion or substitution , ˆ 𝑏 differs from 𝑏 in different ways.
Without loss of generality, let 𝑦 𝐾 denote the true class label of 𝑥 and
all the other 𝑦 𝑘 ( 𝑘 = { 1 ,...,𝐾 − 1 } ) are the potential targets of an evasion attack.
The goal of attack is to make 𝑓 𝑦 𝐾 ( 𝑥, ˆ 𝑏 ) as low as possible and increase 𝑓 𝑦 𝑘 ( 𝑥, ˆ 𝑏 ) of a ﬁxed 𝑘 ( targeted attack ) or any 𝑘 of { 1 ,...,𝐾 − 1 } ( non-targeted attack ) as high as possible simultaneously.
We focus on non-targeted evasion attack , and leave the targeted scenario for future study.
3. I NFORMATION - THEORETIC C HARACTERIZATION OF A DVERSARIAL V ULNERABILITY
Theorem 1 For an instance ( 𝑥,𝑦 ) and a training set 𝑆 sampled from the same underlying distribution 𝜇 𝑥,𝑦 , 𝑓 ∈H is trained using 𝑆 with a deterministic training paradigm.
The expected adversarial risk R 𝑎𝑑𝑣𝜀 deﬁned in Eq.1 can be bounded as in Eq.2, if the loss function ℓ in R 𝑎𝑑𝑣𝜀 adopts the zero-one loss.
[Equation]
where const denotes a constant term.
diff ( 𝑏, ˆ 𝑏 ) indicates the set of the categorical attributes modiﬁed in the attack.
𝐼 ( 𝑥 ; 𝑦,𝑆 ) denotes the mutual information between the feature 𝑥 and the pair of label 𝑦 and training set 𝑆 .
𝜂 𝑥 is the supremum of the difference between the mutual information before and after adversarial perturbation, noted by 𝐼 ( 𝑥 ; 𝑦,𝑆 ) and 𝐼 ( ˆ 𝑥 ; 𝑦,𝑆 ) respectively.
𝐼 ( 𝑓 𝑦 ; 𝑆 ) denotes the mutual information between 𝑆 and 𝑓 .
Theorem 1 unveils the three impact factors jointly determining adversarial vulnerability of the targeted classiﬁer 𝑓 (for answering Q1 in Section 1).
The proof is given in Appendix.
A.
In Eq.2, a lower mutual information 𝐼 ( 𝑥 ; 𝑦,𝑆 ) indicates higher adversarial risk , i.e., R 𝑎𝑑𝑣𝜀 has a higher lower bound.
A lower 𝐼 ( 𝑥 ; 𝑦,𝑆 ) denotes weaker consistency between the input instance ( 𝑥,𝑦 ) and the training set 𝑆 .
The classiﬁer trained by 𝑆 thus produces weaker decision conﬁdence for such ( 𝑥,𝑦 ) .
With ( 𝑥,𝑦 ) dropped to the ambiguous zone near the classiﬁcation boundary, the classiﬁer is more prone to adversarial perturbation.
A higher 𝐼 ( 𝑓 𝑦 ; 𝑆 ) leads to a higher adversarial risk according to Eq.2 .
The mutual information 𝐼 ( 𝑓 𝑦 ; 𝑆 ) reﬂects the dependence between the classiﬁer 𝑓 and the training set 𝑆 , which can be considered as a lower bound of the VC-dimension for countable hypothesis space 𝑓 ∈ H (Xu & Raginsky, 2017;
Zhu et al., 2020).
A higher 𝐼 ( 𝑓 𝑦 ; 𝑆 ) thus denotes that 𝑓 is more likely to suffer from being overﬁtted to 𝑆 .
Model overﬁting is one of the causes of adversarial vulnerability (Tu et al., 2019).
Resonating with the association between 𝐼 ( 𝑓 𝑦 ; 𝑆 ) and adversarial risk, three popularly used robustness enhancement methods controlling the classiﬁer’s complexity and overﬁting risk can potentially adjust the adversarial vulnerability of the classiﬁer with categorical inputs.
Adversarial training (Miyato et al., 2016;
Sinha et al., 2018; Florian et al., 2018; Wang et al., 2019; Shafahi et al., 2019).
The adversarially retrained classiﬁer ˆ 𝑓 is less correlated with the original training set 𝑆 compared to 𝑓 , which reduces 𝐼 ( 𝑓 𝑦 ; 𝑆 ) and may help mitigate the adversarial threat.
The addition of nuclear norm regularization over the classiﬁer’s parameters in the training process (Ravi et al., 2019; Tu et al., 2019).
The resultant regularized classiﬁer has a controlled model complexity, which can potentially reduce the adversarial risk.
Random smoothing (Lee et al., 2019; Levine & Feizi, 2020; Dvijotham et al., 2020; Boj.
et al., 2020).
Following (Cohen et al., 2019), this defense method randomly selects and ﬂips the input categorical features of the targeted classiﬁer.
The randomly perturbed classiﬁcation output is less correlated with the training data 𝑆 ’s distribution, which may reduce adversarial risk.
A higher 𝜂 𝑥 in Eq.3 indicates a higher loss of predicative information by changing 𝑥 to ˆ 𝑥 , and thus a higher adversarial risk .
In this sense, the vulnerability of the classiﬁer depends on the sensitivity of the attacked categorical features with respect to the classiﬁcation task.
3. Scoring Robustness via Set Function Optimization
We formulate the assessment of adversarial robustness level on categorical data as a set function maximization problem : we aim at ﬁnding a minimal set of categorical feature perturbations 𝑙 = diff ( 𝑏, ˆ 𝑏 ) , with which the maximum gap between the classiﬁcation conﬁdence on any wrong label 𝑘 and the correct label 𝐾 is larger than a predeﬁned threshold Γ .
The size of the minimal perturbation set 𝑙 is used to assessment adversarial robustness of 𝑓 over the input instance.
Within the perturbation budget 𝜀 , a smaller/larger | 𝑙 | provoking the alert indicates that the classiﬁer’s output on the input 𝑥 is more/less vulnerable w.r.t.
adversarial perturbation in the categorical feature space.
We consider both the pessimistic (in Eq. 4) and optimistic scenario (in Eq. 5) from the defender’s perspective for robustness scoring. [Equation]
[Equation]
where 𝑚 𝑓 = max 𝑘 ∈{ 1 ,...,𝐾 − 1 } { 𝑓 𝑦 𝑘
( 𝑥, ˆ 𝑏 )}− 𝑓 𝑦 𝐾 ( 𝑥, ˆ 𝑏 ) is the gap deﬁned with an input instance 𝑥 .
𝑚 𝑓 ∈ [− 1 , 0 ) and 𝑚 𝑓 ≥ 0 correspond to correct and wrong classiﬁcation respectively.
Γ is a threshold reﬂecting different levels of tolerance to the perturbation of the classiﬁer’s output.
𝑚 𝑓 ≥ Γ then triggers an alert of potential adversarial attack.
Safety-critical domains, such as cyber security, prefer a lower Γ to provoke alerts even with small yet suspicious input perturbations.
Observation 1 With the threshold Γ = 0 , 𝜓 𝑝𝑒𝑠𝑠 ≥ 0 derived by solving the pessimistic assessment problem in Eq.4 produces an empirical estimate of the expected adversarial risk 𝑓 with an input instance ( 𝑥,𝑦 ) sampled from 𝜇 𝑥,𝑦 .
We leave the derivation of this observation in Appendix.
B. Eq.
4 is a bi-level optimization problem.
The inner level seeks the optimal change of the category value of a given categorical feature, in order to produce maximum possible perturbation on the classiﬁer’s output.
The outer level ﬁnds the optimal combination of different categorical features for maximizing 𝑚 𝑓 within the attack budget.
| 𝑙 | is thus a pessimistic robustness assessment with respect to a knowledgeable adversary .
The adversary can locate the most effective modiﬁcation for both individual categorical features and the combination set of the categorical features.
In contrast, | 𝑙 | to Eq.
5 reﬂects an optimistic robustness assessment with respect to an oblivious adversary .
In practices, an adversary can be limited by the knowledge about the targeted classiﬁer and/or the query budget.
Such an oblivious adversary can only locate the effective category values to attack for a given categorical feature, while can’t ﬁnd the optimal feature combinations to solve the outer-level optimization.
3. Provably Accurate Robustness Assessment via Weakly Submodular Maximization
Exactly solving Eq.4 and Eq.5 is still NP-hard.
To answer Q2 in Section 1, Theorem 2 and Theoremelaborate the optimality guarantee of solving both robustness assessment problems via computationally efﬁcient greedy search, for any classiﬁer meeting the smoothness condition in Deﬁnition.2.
Deﬁnition
Smoothness Condition of 𝑓 .
Let Ω
= ( 𝑝,𝑞 ) , 𝑝,𝑞 ∈ R 𝑛 and 𝑓 : R 𝑛
→ R be a Lipschitzcontinuous and differentiable function.
A function 𝑓 is ( 𝑚 Ω ,𝑀 Ω ) -smooth on Ω , if for any ( 𝑝,𝑞 ) ∈ Ω , 𝑚 Ω ∈ R and 𝑀 Ω ∈ R + , 𝜖 = 𝑓 ( 𝑞 )− 𝑓 ( 𝑝 )−(cid:104)∇ 𝑓 ( 𝑝 ) ,𝑞 − 𝑝 (cid:105) satisﬁes:
[Equation]
Different from (Elenberg et al., 2018), Eq.6 controls only the ﬂuctuation of the function value by varying the input from 𝑝 to 𝑞 and vice versa, regardless of convexity or concavity of 𝑓 .
It allows broader choices of the classiﬁer’s architecture.
With 𝛾 -weak submodularity (Elenberg et al., 2018;
Santiago & Yoshida, 2020) deﬁned in Appendix C, we establish Theorem 2.
Theorem 2 Let Ω [Equation] [Equation] ≤
[Equation]
≤ [Equation] ˆ 𝑏 [Equation]
ˆ 𝑏 [Equation] )| ≤ 𝜁,𝜁 ≥ 1 } , where ˆ 𝑏 and ˆ 𝑏 (cid:48) denote two sets of attribute changes. If the classiﬁer 𝑓 𝑦 𝑘 ( 𝑥 ) (k=1,..., 𝐾 ) follows the regularity condition given by ( 𝑚 𝑘, Ω 𝜁 , 𝑀 𝑘, Ω 𝜁 ) -smoothness constraint on Ω 𝜁 , the pessimistic and optimistic robustness assessment (with Eq.4 and Eq.5) can be formulated respectively as monotone and non-monotone 𝛾 𝜁 -weakly submodular maximization .
Let 𝜖 𝑘 = 𝑓 𝑦 𝑘
( 𝑥, ˆ 𝑏 (cid:48) )− 𝑓 𝑦 𝑘 ( 𝑥, ˆ 𝑏 )
−(cid:104)∇ 𝑓 𝑦 𝑘 ( 𝑥, ˆ 𝑏 ) , ˆ 𝑏 (cid:48) − ˆ 𝑏 (cid:105) , and ∇ 𝑓 𝑦 ( 𝑥,𝑏 )
𝜈 denote the elements of ∇ 𝑓 𝑦 ( 𝑥,𝑏 ) corresponding to the difference between ˆ 𝑏 and ˆ 𝑏 (cid:48) , where 𝜈 = diff ( ˆ 𝑏, ˆ 𝑏 (cid:48) ) . The
ratio 𝛾 pess 𝜁 for the pessimistic robustness assessment is bounded as:
[Equation]
(7) where 𝛾 pess 𝑘<𝐾,𝜁 ≥ (cid:107)∇ 𝑓 𝑦𝑘 ( 𝑥,𝑏 ) 𝜈
(cid:107) 2 + 𝑚 𝑘, Ω 1 |
𝜁
|/ 2 (cid:107)∇ 𝑓 𝑦𝑘 ( 𝑥,𝑏 )
𝜈
(cid:107)
+ 𝑀 𝑘, Ω 𝜁 | 𝜁 |/ 2 for 𝜖 𝑘 ≥ 0 , 𝛾 pess 𝑘<𝐾,𝜁 ≥ 2 𝑚 𝑘, Ω 𝜁 (cid:107)∇ 𝑓 𝑦𝑘 ( 𝑥,𝑏 )
𝜈 (cid:107)
((cid:107)∇ 𝑓 𝑦 𝑘 ( 𝑥, 𝑏 ) 𝜈 (cid:107) 2 − 𝑀 𝑘, Ω 1 | 𝜁 |/ 2 ) for 𝜖 𝑘 < 0 , 𝛾 pess 𝐾,𝜁 ≥ 2 𝑚 𝐾, Ω 𝜁 (cid:107)∇ 𝑓 𝑦𝐾 ( 𝑥,𝑏 ) 𝜈
(cid:107)
((cid:107)∇ 𝑓 𝑦 = 𝐾 ( 𝑥,𝑏 )
𝜈
(cid:107) 2 − 𝑀 𝐾, Ω 1 | 𝜁 |/ 2 ) for 𝜖 𝐾 ≥ 0 , and 𝛾 pess 𝐾,𝜁 ≥ (cid:107)∇ 𝑓 𝑦𝐾 ( 𝑥,𝑏 ) 𝜈
(cid:107) 2 + 𝑚 𝐾, Ω 1 | 𝜁 |/ 2 (cid:107)∇ 𝑓 𝑦𝐾 ( 𝑥,𝑏 ) 𝜈
(cid:107)
+ 𝑀 𝐾, Ω 𝜁 | 𝜁 |/ 2 for 𝜖 𝐾 < 0 .
Similarly, the submodularity ratio
𝛾 optim 𝜁 of the optimistic assessment is:
[Equation]
) where 𝛾 optim 𝑘<𝐾,𝜁 ≤ (cid:107)∇ 𝑓 𝑦𝑘 ( 𝑥,𝑏 )
𝜈 (cid:107)2 𝑀 𝑘, Ω 𝜁 ((cid:107)∇ 𝑓 𝑦𝑘 ( 𝑥,𝑏 )
𝜈
{+(cid:107) 2 − 𝑚 𝑘, Ω 1 | 𝜁 |/ 2 ) for 𝜖 𝑘 ≥ 0 , 𝛾 optim 𝑘<𝐾,𝜁 ≤ (cid:107)∇ 𝑓 𝑦𝑘 ( 𝑥,𝑏 ) 𝜈 (cid:107) 2 + 𝑚 𝑘, Ω 𝜁 | 𝜁 |/(cid:107)∇ 𝑓 𝑦𝑘 ( 𝑥,𝑏 ) 𝜈 (cid:107) 
+}
+ 𝑀 𝑘, Ω 1 | 𝜁 |/ 2 for 𝜖 𝑘 < 0 , 𝛾 optim 𝐾,𝜁 ≤ (cid:107)∇ 𝑓 𝑦𝐾 ( 𝑥,𝑏 ) 𝜈 (cid:107) 2 + 𝑚 𝐾, Ω 𝜁 | 𝜁 |/ 2 (cid:107)∇ 𝑓 𝑦𝐾 ( 𝑥,𝑏 ) 𝜈
{+(cid:107) 
+}
+ 𝑀 𝐾, Ω 1 | 𝜁 |/ 2 for 𝜖 𝐾 ≥ 0 , and 𝛾 optim 𝐾,𝜁 ≤ (cid:107)∇ 𝑓 𝑦𝐾 ( 𝑥,𝑏 ) 𝜈 (cid:107) 22 2 𝑀 𝐾, Ω 𝜁 ((cid:107)∇ 𝑓 𝑦𝐾 ( 𝑥,𝑏 ) 𝜈
(cid:107)
2 − 𝑚 𝐾, Ω 1 | 𝜁 |/ 2 ) for 𝜖 𝐾 < 0 .
Enjoying the weak submodularity, the pessimistic and optimistic assessment can be derived by greedy search with provable accuracy.
We present a Forward Stepwise Greedy Search (FSGS) algorithm for deriving the pessimistic assessment (Eq.4).
Due to the non-monotone objective function of the optimistic assessment (Eq.5), a stochastic variant of the standard FSGS , namely Randomized FSGS (RandGS) is adopted, following the suggestion in (Buchbinder et al., 2014).
We give the pseudo codes of FSGS and RandGS in Algorithm.1 and 2 in Appendix.
D. Compared to FSGS , the difference in RandGS is to randomly select one of the top-ranked candidate features contributing with the largest marginal gain 𝑚 𝑓 to the objective function.
Theorem 3 gives the approximation guarantee of the pessimistic and optimistic robustness assessment using FSGS and RandGS respectively.
Theorem 3 For a ( 𝑚 Ω 𝜁 ,𝑀 Ω 𝜁 ) -smooth classiﬁer 𝑓 , the approximation quality of the solution 𝑙 to Eq.
and Eq.5 is bounded respectively as
[Equation]
where 𝛾 pess 𝜁 and 𝛾 optim 𝜁 are the submodularity ratio in Eq.
7 and Eq. 8, respectively.
𝑙 ∗ 𝑝𝑒𝑠𝑠 and 𝑙 ∗ 𝑜𝑝𝑡𝑖𝑚 are the true optimal solutions to the two problems.
With Theorem 3, the Q2 in Section.1 is fully addressed.
Notably, any classiﬁer with a ﬁnite Lipshitz constant meets the smoothness condition.
For practical estimation, 𝑀 Ω 𝜁 can be computed as local
Lipshitz constant of 𝑓 if 𝜖 > 0 in Deﬁnition.2 (or − 𝑓
( 𝑥 ) if 𝜖 < 0 ).
Furthermore, 𝑚 Ω 𝜁 can be calculated as the local strong convexity constant of 𝑓 ( 𝑥 ) if 𝜖 > 0 in Deﬁnition.2 (or − 𝑓 ( 𝑥 ) if 𝜖 < 0 ).
Recent works (Fazlyab et al., 2019; Jordan & Dimakis, 2021) have shown that a large number of neural networks are Lipschitz continuous, such as the fully-connected neural networks with ReLU , sigmoid , and tanh activation functions.
The FSGS based robustness assessment method can thus be applied to most
DNN-based classiﬁers that are popularly adopted in real-world applications.
Theorem 3 also reveals that the role that the smoothness of 𝑓 𝑦 plays in characterizing the solvability of the robustness assessment problem .
As indicated by Eq.
7 and Eq. 8, a smoother 𝑓 𝑘 with smaller 𝑚 𝑘, Ω 𝜁
and 𝑀 𝑘, Ω 𝜁 has a higher submodularity ratio ( 𝛾 pess 𝑘,𝜁 and 𝛾 optim 𝑘,𝜁 ).
With a submodularity ratio closer to 1, both FSGS and RandGS can obtain better approximation quality to the underlying optimal result of the robustness assessment, according to Eq. 9.
Comparing Eq.
7 to 8, the submodularity ratio of the pessimistic assessment is higher than that of the optimistic opponent.
The greedy search has lower approximation quality under the optimistic setting.
The optimistic assessment is thus generally higher than the pessimistic assessment.
It echos with the intuition that a classiﬁer is more vulnerable to a knowledgeable adversary than an oblivious one.
3. O RTHOGONAL M ATCHING P URSUIT G REEDY S EARCH FOR R OBUSTNESS A SSESSMENT
Both FSGS and RandGS need to traverse all the combinations of candidate attributes and the subsets of the selected set of attributes.
In the worst case, they require (cid:205)
𝑇𝑡 =
(( 𝑛 − 𝑡 )∗ 𝑚 ∗ 2 𝑡 ) objective function evaluations for 𝑇 iterations, where 𝑛 denotes of the number of categorical features in an instance 𝑥 and each feature can choose any of the 𝑚 categorical values.
Large 𝑛 and 𝑚 are usually witnessed in the real-world data.
Both methods thus are costly in querying the oracle classiﬁer in such real-world cases.
In (Wang et al., 2020), an orthogonal matching pursuit guided greedy searching strategy ( OMPGS ) is introduced to address the query bottleneck of primary greedy search in evasion attack on discrete data.
In the context of attack, this method computes ∇ 𝑓 𝑦 ( 𝑥 ) ˆ 𝑏 (the gradient of 𝑓 𝑦 w.r.t.
the binary indicators ˆ 𝑏 of unmodiﬁed categorical features).
The magnitude of the gradient measures the capability of each candidate feature in reducing the classiﬁcation conﬁdence of the correct label.
In this sense, OMPGS narrows down the candidate attributes to those that best correlate with the orthogonal complement of the subset of the attributes already selected in greedy search.
The query complexity of OMPGS is thus independent of 𝑛 and 𝑚 .
Due to the space limit, the algorithm implementation of OMPGS are explained in Algorithm.3 in Appendix.
E.
We apply OMPGS to speed up the solutions to both the pessimistic and optimistic assessment problems.
The approximation guarantee of OMPGS given in (Wang et al., 2020) only explains the provably accuracy of using OMPGS in the pessimistic assessment .
In this work, we further bound the approximation accuracy of OMPGS to derive the optimistic assessment in Theorem 4.
Comparing to Theorem 3, OMPGS achieves a lower but similar approximation accuracy to the underlying optimal solution to the optimistic assessment (Eq. 5).
Theorem 4 Following the notations in Theorem 2 and Theorem.3, the approximation quality of the optimistic assessment score | 𝑙 | obtained by OMPGS is bounded as in the followings:
[Equation]
In both of the pessimistic and optimistic assessment, the gradient magnitude used by OMPGS can be interpreted as an upper bound of the increase of 𝑚 𝑓 by adding adversarial perturbation to a previously unmodiﬁed feature, as shown in Appendix E. Notably, the gradient used in OMPGS is not taken directly with respect to the categorical features.
Instead, the gradient is taken with respect to the binary indicator variable 𝑏 𝑗𝑖 (see Section.3).
We rank the categorical features according to the magnitudes of the gradients with respect to different 𝑏 𝑗𝑖 .
A higher gradient magnitude indicates that the corresponding categorical feature 𝑥 𝑖
= 𝑥 𝑗𝑖 can contribute higher marginal gain of 𝑚 𝑓 .
OMPGS narrows down range of greedy search.
Only the top ranked candidate features are considered to modify.
[Table]
E XPERIMENTAL S TUDY
We instantiate the study with standard LSTM based classiﬁers trained on three multi-class datasets collected from real-world applications of Text analysis , Cyber security and Biomedicine .
Yelp-5 ( Yelp )(Asghar, 2016).
The Yelp-5 reviews dataset was obtained from the Yelp Dataset Challenge in 2015.
We use the reviews containing 650K training and 50K testing samples with the classes from 1 star to 5 stars for learning classiﬁer.
Each word is a categorical feature with 300-dim embedding.
Intrusion Prevention System Dataset (IPS) (Wang et al., 2020).
The IPS dataset has 242, instances, each of which is a 20-step attack sequence and represented by 𝑥 ∈ R 20 ∗ 1103 ∗ 70 , because each step can be a categorical value from 1,103 different malicious actions embedded in 70-dim space by word embedding.
An 𝑥 is classiﬁed with 3 labels: attack type 1, attack type 2 , and others .
Splice-junction Gene Sequences (Splice) (Noordewier et al., 1991).
This dataset has 3,190 samples.
Each one is a gene fragment of 60 categorical features with values in { ‘A’, ‘G’, ‘C’, ‘T’, ‘N’ } .
Learning an embedding for each value with 30-dim, one instance is given as 𝑥 ∈ R 60 ∗ 5 ∗ 30 .
The task is to predict each splice instance as an intro-exon boundary ( IE ), an exon-intro boundary ( EI ), or neither of them.
The LSTM-based classiﬁers with ReLu activation function and dropout achieve accuracy scores of 0.61 , 0.92 and 0.95 respectively for Yelp , IPS and Splice .
More details about the datasets and the experimental setting are given in Appendix F.1.
To strengthen the universality of the robustness assessment framework, we also include additional results on binary-classiﬁcation datasets using
Convolution Neural Networks (CNN) in Appendix.
F.
The experimental study is to address questions below:
1) What’s the robustness assessment result of a targeted classiﬁer?
(Empirically evaluate the answer in Theorem 2,3 and 4 to Q2 in Section 4.1).
For the classiﬁer trained on each dataset, we ﬁrst ﬁnd the robustness assessment | 𝑙 | for each data instance.
We report the median and mean value of all the | 𝑙 | derived on the dataset.
A lower/higher median value indicates a weaker/stronger robustness of the classiﬁer to the adversarial attack over the given data set.
The tolerance threshold Γ is tested on − 0 .
4 and 0 to assess our proposed assessment method with varied tolerance to adversarial threats in safety-critical applications.
Different greedy search methods are compared on their assessment result.
Besides FSGS and OMPGS (for solving Eq.4) and RandGS and RandOMPGS (for solving Eq.5) , we also include Gradient-based Attack (GradAttack) from (Qi et al., 2019) in the evaluation.
GradAttack tackles only the pessimistic assessment problem.
The comparison with GradAttack is to verify the merit of FSGS and its variants in solving the robustness assessment problem of a general classiﬁer.
For
RandGS and OMPGS , we empirically choose 10 candidate features in each iteration of greedy search 1 .
) Can the information-theoretic characterization of adversarial vulnerability in Theorem 1 be veriﬁed using the proposed robustness assessment
Empirically evaluate the answer
in Theorem to Q in Section .2 and 4.
We verify an increase of risk on 𝑥 𝑙 𝑚𝑖 with lower mutual information i
e
, 𝐼 𝑥 𝑙 𝑚𝑖 ; 𝑦,𝑆 ) < 𝐼 ( 𝑥 ; 𝑦,𝑆 and verify an increase of robustness by reducing 𝐼 ( 𝑓 𝑦 ; 𝑆 ) .
We compare the time complexity of FSGS , OMPGS and GradAttack .
Furthermore, we evaluate the averaged running time and the averaged iterations of the three methods over Yelp , IPS and Splice datasets.
As shown, both FSGS and OMPGS stop early the search with less loops than GradAttack .
OMPGS costs much fewer queries and less running time than FSGS .
Compared to GradAtttack ,
OMPGS has a better chance to select the really useful feature modiﬁcations, thus costing fewer loops.
We provide the results in Appendix.
F.3 due to the space limit.
4. R OBUSTNESS A SSESSMENT W ITH G REEDY S EARCH
Besides showing the median and mean of the assessment score | 𝑙 | ( Med(Avg )) of the testing instances, we also report attack success rate (noted as SR in the tables), i.e., the fraction of the testing instances on which the greedy methods can push 𝑚 𝑓 greater than the tolerance threshold within the one-hour time limit.
With similar median/mean values of | 𝑙 | , a lower SR also indicates stronger robustness of the classiﬁer.
If the greedy search fails to push 𝑚 𝑓 to surpass the threshold Γ for all the testing instances within the time limit, we note this situation as OVF (an abbreviation for overﬂow) and SR = 0 in the tables.
𝑃 / 𝑂 denotes the pessimistic or optimistic assessment respectively.
Table 1 shows several interesting ﬁndings.
First, for both the pessimistic and optimistic robustness assessment, a lower tolerance threshold Γ results in lower scores over all the datasets .
This observation is consistent with the intuition of setting the tolerance threshold.
A lower tolerance threshold helps choose more worst-case resilient classiﬁers.
Our study allows for the analysis with different vigilance levels.
Second, as expected, the pessimistic robustness assessment scores are always signiﬁcantly lower than those of the optimistic assessment, and the SR value is much lower on the optimistic assessment scenario.
Due to lack of query capability and knowledge about 𝑓 , it is difﬁcult for the oblivious adversary to locate the sensitive features.
OMPGS and RandOMPGS produce similar robustness assessment scores, compared to FSGS and
RandGS .
Compared to evaluate each candidate features as in FSGS / RandGS , OMPGS improves the search efﬁciency using the gradient based heuristic guidance.
On IPS and Splice , FSGS / RandGS needs 217.65 and 1123.51 seconds on average on each data sample for the pessimistic and optimistic assessment.
In contrast, OMPGS / RandOMPGS costs only 0.24 and 1.59 seconds for the pessimistic and optimistic assessment, more than 700 times faster than FSGS / RandGS .
GradAttack , in contrast, produces much lower SR than FSGS and OMPGS in the pessimistic assessment setting.
As discussed in
Section.2, the greedy search in GradAttack ignores the combinatorial effect of the candidate features and the previously modiﬁed features, which leads to worse approximation quality and much less efﬁcient attacks compared to FSGS and OMPGS .
The results echo the theoretical analysis.
With signiﬁcantly lower SR , GradAttack can not provide an accurate answer to the robustness assessment problem.
We conduct one-factor-at-a-time sensitivity analysis on each dataset (Campbell et al., 2008) to measure the sensitivity of features in the classiﬁcation task.
Given a data instance, we change each feature while keeping all the others ﬁxed.
The averaged change of the classiﬁer’s output over all the testing instances is used as the feature-wise sensitivity measurement.
A larger value indicates that the classiﬁer’s output is more sensitive to the change over the corresponding feature.
We ﬁnd that the top-sensitive features also appear as the most frequently selected features by FSGS , GradAttack and OMPGS .
Details about the sensitivity analysis can be found in Appendix.
F.4.
The interesting overlapping between the attributes which are useful for attack and the top-ranked sensitive attributes conﬁrms our intuition about the association between feature sensitivity and adversarial vulnerability of the classiﬁer.
4. R OBUSTNESS VS F EATURE I NFORMATIVENESS
According to the discussion in Theorem 1, a lower 𝐼 ( 𝑥 ; 𝑦,𝑆 ) indicates a higher adversarial risk on the input instance ( 𝑥,𝑦 ) , implying a smaller robustness assessment score.
To verify such impact of feature informativeness, we compare the robustness on 𝑥 and on 𝑥 𝑙 𝑚𝑖 that has a lower mutual information, i.e., 𝐼 ( 𝑥 𝑙 𝑚𝑖 ; 𝑦,𝑆 ) <
𝐼 ( 𝑥 ; 𝑦,𝑆 ) .
The sample 𝑥 𝑙 𝑚𝑖 is constructed as those on the half-way of pushing the 𝑚 𝑓 of the original sample 𝑥 to surpass Γ .
For each dataset, we select the instances requiring at the least two modiﬁed features in order to deliver a successful attack.
These instances are treated as original .
There are 125, 414 and 324 original instances in Yelp , IPS , and Splice respectively.
The results in
Table 2 conﬁrm that the classiﬁer becomes less robust to adversarial modiﬁcations over 𝑥 𝑙 𝑚𝑖 samples in the ambiguous zone, i.e., higher SR and smaller | 𝑙 | on the 𝑥 𝑙 𝑚𝑖 samples than the original samples 𝑥 .
4. R OBUSTNESS WITH DEFENSIVE TECHNIQUES
As discussed in Theorem.1, reducing 𝐼 ( 𝑓 𝑦 ; 𝑆 ) , the dependence of a classiﬁer over the training set 𝑆 , improves the classiﬁer’s adversarial robustness.
We tune a classiﬁer via adversarial training ( AdvC ), the nuclear-norm based regularization ( NuR ) following (Ravi et al., 2019) and random smoothing by ( RS ) as suggested in (Boj.
et al., 2020) to reduce 𝐼 ( 𝑓 𝑦 ; 𝑆 ) , and then evaluate if the pessimistic robustness assessment becomes higher, e..g, descrese of SR.
To adopt adversarial training, we solve Eq.
4 using
[Table] [Table]
OMPGS until the value of 𝑚 𝑓 drops into the interval [− 0 .
The resultant modiﬁed instances are used for adversarial training.
The regularization parameter of the nuclear-norm regularized training method is chosen following (Ravi et al., 2019).
The random smoothing is conducted by randomly ﬂipping the values of input categorical features ( RS ) with 𝑝 − = 0 .
6 and 𝑝 + = 0 .
01 as suggested in (Boj.
et al., 2020).
We also implement the original sparsity-aware random smoothing based certiﬁable robustness ( SRS ) (Boj.
et al., 2020) as a reference method of random smoothing based defense techniques.
It performs ﬁrst the same randomly ﬂipping of categorical features as in RS .
The least number of the features changed to trigger a wrong classiﬁcation, a.k.a.
certiﬁable robustness , is then estimated via a dimension-independent evaluation based on the randomly ﬂipped features.
SRS (Boj.
etal.
,2020)isreportedtobesigniﬁcantlymoreefﬁcientandproducetighterrobustnessassessmentsthan other random smoothing-based methods (Lee et al., 2019; Levine & Feizi, 2020; Dvijotham et al., 2020).
Table 3 illustrates the pessimistic robustness assessment scores of the tuned classiﬁers with the tolerance threshold Γ= 0 .
We skip the results of the optimistic robustness assessment and the results with Γ= − 0 .
4 as they present a similar variation tendency.
Both adversarial training and nuclear-norm regularization help improve the classiﬁer’s robustness, leading to lower SR and higher | 𝑙 | over all the three datasets in most of the cases.
However, on Splice , the adversarial vulnerability reduction effect is much less visible than those observed on the other two datasets.
We ascribe this phenomenon to the fact that the feature-wise sensitivity on Splice has a more skewed distribution than the other datasets.
A few categorical features in Splice are signiﬁcantly more sensitive than the others in the classiﬁcation task, corresponding to important proteins dominating the molecules’ properties (see feature sensitivity analysis in Section 4.1 and Appendix.
F.4).
The existence of such highly sensitive features makes it difﬁcult to enhance the targeted classiﬁer’s robustness without losing the classiﬁcation utility.
RS and
SRS have higher SR than the other robustness enhanced models, showing that the random smoothing technique helps the least improve robustness or even make it more vulnerable to the attack, in the case where a fraction of categorical features are highly sensitive.
As seen in Table.1, highly sensitive features can be witnessed in many diversiﬁed applications.
Firstly, ﬂipping these sensitive features can lead to miss-classiﬁcation rather than providing robustness.
Secondly, if random ﬂipping misses the sensitive features, the adversary can still target at these features to deliver a successful attack.
C ONCLUSION
We unveil the key factors that jointly determine the adversarial vulnerability of a classiﬁer with categorical inputs via information-theoretic analysis.
The characteristics of the categorical data and the functional property of the targeted classiﬁer act as the external and internal cause of the adversarial threat.
We further set a theoretical association between the smoothness of the targeted classiﬁer and the solvability of both pessimistic and optimistic robustness assessment with categorical inputs.
Despite of the NP-hard nature of the problem, we show that a provably accurate assessment is approachable for any classiﬁer using simple yet efﬁcient greedy search.
Our study is useful for assessing the model robustness before deploying it to real-world safety-critical applications.
Our future work will pursue to provide tighter quality bound for assessing adversarial robustness assessment on categorical data. A CKNOWLEDGEMENTS
The research reported in this publication was partially supported by funding from King Abdullah
University of Science and Technology (KAUST)
A P ROOF FOR T HEOREM
Lemma 1 Supposing 𝐴 and 𝐵 are two independent random variables and U is given by 𝑃 𝑈𝐴𝐵
[Equation]
𝐴𝐵
[Equation]
Proof: Given the deﬁnition of mutual information 𝐼 ( 𝑋 ; 𝑌 )
=
𝐻 ( 𝑋 )− 𝐻 ( 𝑋 | 𝑌 ) , we need to prove:
[Equation]
With the chain rule of conditional entropy:
[Equation]
It indicates that 𝐻 ( 𝐴,𝐵 | 𝑈 ) ≤ 𝐻 ( 𝐴 | 𝑈 ) , as 𝐻 ( 𝐵 | 𝑈 ) ≥ 0 .
Similarly, we can derive 𝐻 ( 𝐴,𝐵 | 𝑈 ) ≤ 𝐻 ( 𝐵 | 𝑈 ) as well.
Combing both inequality relations, Eq.12 holds, which in turn proves Eq.11.
Considering the markov chain in Figure.1, we assume a class label Y and the training set 𝑆 are statistically independent.
𝑓 depends on S as 𝑓 =
A( 𝑆 ) , where A is the training paradigm (could be randomized).
The generation of the adversarial example 𝑋 (cid:48) depends on both the clean feature vector 𝑋 and the targeted classiﬁer 𝑓 .
This is consistent with the fact that generating adversarial samples needs to understand the local geometry property of the decision boundary of the targeted classiﬁer.
Once we get the perturbed feature vector 𝑋 (cid:48) , the classiﬁcation output 𝑌 (cid:48)
=
𝑓 ( 𝑋 (cid:48) ) .
We can derive the lower bound of the miss-classiﬁcation probability with the perturbed categorical input 𝑥 (cid:48)
following Fano inequality:
[Equation]
[Equation]
where 𝐾 denotes the number of the classes in the learning task.
Based on Data Processing Inequality illustrated in Chapter 2 (Mack, 2006), we can further obtain:
[Equation]
where 𝜂 𝑋 =
𝐼 ( 𝑋 ; 𝑌,𝑆 )− 𝐼 ( 𝑋 (cid:48) ; 𝑌,𝑆 ) denotes the mutual information difference between the unperturbed 𝑋 and the adversarially perturbed instance 𝑋 (cid:48) .
By applying Lemma.1 on 𝐼 ( 𝑋 (cid:48) ; 𝑌,𝑆 ) in Eq.15, we have
[Equation]
DataProcessingInequality
[Equation]
B E XPLANATION TO O BSERVATION .
First , we follow the IID assumption in the deﬁnition of the expected adversarial risk in Eq.1.
We assume ℓ adopts the form of zero-one loss function.
If 𝑓 ( ˆ 𝑥 ) ≠ 𝑦 , ℓ ( 𝑓 ( ˆ 𝑥 ) ≠ 𝑦 )) = 1 .
Otherwise, ℓ = 0 .
With this setting of the loss function, the expected adversarial risk of the classiﬁer 𝑓 can be formulated as in Eq.17.
This gives Eq.1 (the adversarial risk of 𝑓 ) in the main paper.
[Equation]
17) where 𝑃 denotes the probability measurement.
In this sense, the expected adversarial risk of R 𝑎𝑑𝑣𝜀 can be explained intuitively as the maximum miss-classiﬁcation probability over an adversarial sample ˆ 𝑥 , with the underlying data distribution 𝜇 𝑥,𝑦 and the attack budget limited by 𝜀 .
Second , Theorem.1 provides the information-theoretical lower bound of R 𝑎𝑑𝑣𝜀 with respect to the data distribution 𝜇 𝑥,𝑦 .
Indeed, for any given training data set 𝑆 sampled from the joint distribution 𝜇 𝑥,𝑦 ,
𝑆 can be considered as a random variable.
Similarly, given
a deterministic training paradigm A ,
the classiﬁer 𝑓 =
A ( 𝑆 ) (trained with 𝑆 using the training algorithm A ) can
be
also treated as a random variable in the hypothesis space. For a testing input ( 𝑥 𝑦 ) sampled
from
the
same distribution 𝜇 𝑥,𝑦 as 𝑆 deriving Theorem.1 follows Fano’s inequality in information theory (Chapter 2 in (Mack, 2006))
: • For any 𝑥 ∼ 𝜇 𝑥 and ( 𝑥,𝑦 ) ∼ 𝜇 𝑥,𝑦 , following Fano’s inequality, we can produce the lower bound of miss-classiﬁcation probability over the adversarial sample ˆ 𝑥 generated from 𝑥 (based on the information of 𝑦 and 𝑓 ):
[Equation]
where 𝑓 ( 𝑥 ) =
𝑦 .
The lower bound of the miss-classiﬁcation probability is composed by three mutual information measurements among the random variables ( 𝑥,𝑦 ) , 𝑆 and 𝑓
. • By deﬁnition, the following inequality holds:
[Equation]
where ( 𝑥,𝑦 ) ∼ 𝜇 𝑥,𝑦 , 𝑆 ∼ 𝜇 𝑥,𝑦 and 𝑓 ∈H .
From this perspective, Theorem.1 establishes a lower bound of the expected adversarial risk of 𝑓 :
Given a deterministic training paradigm A , for any training data set 𝑆 and any testing input instance ( 𝑥,𝑦 ) sampled from the underlying data distribution 𝜇 𝑥,𝑦 , the miss-classiﬁcation probability of 𝑓 over the adversarial sample generated from 𝑥 (based on 𝑦 and 𝑓 ) can be bounded by the three mutual information measurements 𝐼 ( 𝑥 ; 𝑦,𝑆 ) , 𝜂 𝑥 and 𝐼 ( 𝑓 ; 𝑆 ) , as in Eq.2 and 3 of Theorem.1 in the main paper.
Comparison with the adversarial risk bound proposed by (Tu et al., 2019) .
A relevant work of adversarial risk analysis can be found in (Tu et al., 2019), where the expected adversarial risk of a classiﬁer 𝑓 is measured by the maximum classiﬁcation loss over the distribution of adversarial samples (Deﬁnition.1 in (Tu et al., 2019)):
[Equation]
(20) where 𝑁 ( 𝑥 ) denotes the neighbor set of 𝑥 derived by modifying features in 𝑥 .
The coverage of the neighboring set 𝑁 ( 𝑥 ) can be measured by 𝑙 𝑝 distance for continuous data or 𝑙 0 norm for discrete data.
After that, Tu et al’s work achieves two results.
First, the expected adversarial risk bound can be bounded from above by the local worst-case risk of 𝑓 (noted as R 𝜀,𝜇 𝑥,𝑦 ):
[Equation]
(21) where 𝜇 ˆ 𝑥 is a distribution sampled from the 𝑝 -Wassernstein ball ( 𝑝 ≥ 1 ) centered at 𝜇 𝑥 and with a radius of 𝜀 .
𝑁 ( 𝑥 ) ⊆
𝜇 ˆ 𝑥 .
Second, the expected adversarial risk R 𝑎𝑑𝑣𝜀 can be bounded by the upper bound of the local worst-case risk of 𝑓 consisting of the rademarcher complexity of 𝑓 and the empirical risk of 𝑓 , according to Theorem.1 and Remark.3 in (Tu et al., 2019).
This upper bound holds with probability at least 1 − 𝛿 :
[Equation] [Equation]
where ˆ R 𝑎𝑑𝑣𝑃 𝑛 denotes the adversarial empirical risk calculated using 𝑛 IID samples from the adversarial data distribution 𝜇 ˆ 𝑥 .
Practically, ˆ R 𝑎𝑑𝑣𝑃 𝑛 can be estimated by calculating the maximum classiﬁcation loss reached by perturbing 𝑥 within the attack budget.
𝑀 denotes the maximum output value of 𝑓 .
(cid:60) 𝑛 ( 𝜓 ( 𝑓 )) denotes the expected rademarche complexity of an augmented function deﬁned on 𝑓 .
The value of (cid:60) 𝑛
( 𝜓 ( 𝑓 )) is independent from the empirical adversarial risk
ˆ R 𝑎𝑑𝑣𝑃 𝑛 .
The formulation of (cid:60) 𝑛
( 𝜓 ( 𝑓 )) can be found on Page 10 of (Tu et al., 2019).
Our work differs from the results in (Tu et al., 2019) from the following perspective.
The core contribution of (Tu et al., 2019) is to associate the Rademacher complexity of the classiﬁer 𝑓 with 𝑓 ’s expected adversarial risk.
Beyond that, for a classiﬁer adopted in a learning scenario, we believe both the characteristics of input data and the classiﬁer’s complexity measurement determine its adversarial risk.
As shown by the mutual information-based lower bound (Eq.2 and 3 of Theorem.1), the mutual information 𝐼 ( 𝑓 ; 𝑆 ) measures the dependence of a trained 𝑓 over the training data 𝑆 sampled from 𝜇 𝑥 𝑦 .
𝐼 ( 𝑓 ; 𝑆 ) can be considered as the lower bound of the VC-dimension of 𝑓 (Xu & Raginsky, 2017; Zhu et al., 2020) which encodes the classiﬁer’s complexity.
The other two mutual information measurements 𝐼 ( 𝑥 ; 𝑦,𝑆 ) and 𝜂 𝑥 relates to the characteristics of the input data ( 𝑥,𝑦 ) .
Intuitively 𝐼 ( 𝑥 ; 𝑦,𝑆 ) measures the informativeness of 𝑥 for classiﬁcation.
A lower 𝐼 ( 𝑥 ; 𝑦,𝑆 ) denotes weaker consistency between the input data instance ( 𝑥,𝑦 ) and the training data set 𝑆 𝜂 𝑥 reﬂects the sensitivity of the modiﬁed categorical features during the evasion attack.
A classiﬁer with input data containing more sensitive features is more likely to be compromised by modifying intentionally the values of the sensitive feature dimensions.
Relations to the proposed robustness measurement.
Since it is not feasible to directly compute the expected risk R 𝑎𝑑𝑣𝜀 , we empirically evaluate the adversarial risk/robustness and presented the results in
Table 2 and Table 3 in Section 4.2 and 4.3.
The empirical evaluation of the adversarial risk evaluation of 𝑓 over the data distribution 𝜇 𝑥,𝑦 is computed as a sample average,
[Equation]
[Equation]
To facilitate computing, we adopt a smooth loss ℓ as a surrogate function approximating the zero-one loss.
In Eq.(23), max ˆ 𝑥 𝑖 , | diff ( ˆ 𝑥 𝑖 ,𝑥 𝑖 )|≤ 𝜀 ℓ ( 𝑓 ( ˆ 𝑥 𝑖 ) ,𝑦 𝑖 ) is to estimate the maximum classiﬁcation loss of 𝑓 over the adversarial sample ˆ 𝑥 𝑖 , which is generated by modifying at most 𝜀 categorical features of 𝑥 𝑖 .
By its nature of set function maximization, ˆ R 𝑎𝑑𝑣 ( 𝑓 ) is calculated in two difference settings: the pessimistic and optimistic robustness assessment deﬁned in Eq.4 and Eq.5 in the main paper.
When setting the threshold Γ = 0 in Eq.4 in the main paper, the pessimistic estimate 𝜓 𝑝𝑒𝑠𝑠 (deﬁned by 𝑓 𝑦 𝑘 − 𝑓 𝑦 𝐾 ) is proportional to the classiﬁcation loss, and gives the miss-classiﬁcation conﬁdence of the target classiﬁer at 𝑥 with the attack budget limited by 𝜀 .
this “miss-classiﬁcation conﬁdence” is the empirical estimator of the adversarial risk 𝑓 has a given input 𝑥 , or called robustness .
C P ROOF FOR THEOREMS ON THE WEAK SUBMODULARITY
We introduce the notion of 𝛾 -weak submodularity used in Theorem 2.
Deﬁnition 3 Monotone and 𝛾 -weakly submodular function (Santiago & Yoshida, 2020).
A set function 𝑔 ( 𝑆 ) is monotone, if for every 𝑇 ⊂ 𝑆 , we have 𝑔 ( 𝑆 ) ≥ 𝑔 ( 𝑇 ) .
Given a set cardinality threshold 𝑘 ≥ 1 and a scalar value 𝛾 ∈ [ 0 , 1 ] .
We deﬁne 𝛾 𝑘 of 𝑔 ( . ) w.r.t.
a set 𝑆 as:,
[Equation]
[Equation]
If 𝑔 ( 𝑆 ) is monotone and 𝛾 𝑘 > 𝛾 for any 𝑘 , 𝑔 ( 𝑆 ) is a 𝛾 -submodular function .
Otherwise, 𝑔 ( 𝑆 ) is 𝛾 -submodular , if 𝑔 ( 𝑆 ) is non-monotone but 𝑔 ( 𝑆 ) follows for any 𝑘 :
[Equation]
It is easy to ﬁnd, if 𝑔 ( 𝑆 ) is monotone and 𝛾 ≥ 1 , 𝑔 ( 𝑆 ) is submodular.
Proof of deriving the robustness assessment score under the pessimistic scenario in Eq.7: we ﬁrst formulate pessimistic scenario as a set function optimization,
[Equation]
[Equation]
where 𝑏 𝑙 denotes the modiﬁcation of categorical features indicated by the index set 𝑙 .
˜ 𝑓
𝐾 ( 𝑏 𝑙 ) = − 𝑓 𝐾 ( 𝑏 𝑙 ) .
For each 𝑘 ∈ 1 , 2 , 3 ,...,𝐾 , 𝜖 𝑘 = 𝑓 𝑘 ( 𝑞 ) − 𝑓 𝑘 ( 𝑝 ) −(cid:104)∇
𝑓 𝑘 ( 𝑝 ) ,𝑞 − 𝑝 (cid:105) .
𝐹 𝑘 ( 𝑆 ) and 𝐺 𝐾
( 𝑆 ) are monotonically non-decreasing set functions.
Supposing 𝜖 𝑘 ≥ 0 for each 𝑘 ∈ { 1 , 2 , 3 , ..., 𝐾 − 1 } , we assume further that the features indicated by 𝑙 (cid:48) are modiﬁed in addition to 𝑙 , with | 𝑙 (cid:48) | ≤ 𝜁 .
For any 𝑏 𝑙 , 𝑏 𝑙 ∪ 𝑙 (cid:48) and 𝑏 𝑙 ∪ 𝑗 denote the additional modiﬁcation over the features indicated by 𝑙 (cid:48) and by 𝑗 respectively to increase the output of 𝑓 𝑘 (increasing the miss-classiﬁcation decision conﬁdence) and − 𝑓 𝐾 (decreasing the conﬁdence of correct classiﬁcation).
In the following analysis, we relax the discrete indicator 𝑏 to the continuous domain, as each 𝑏 𝑖 ∈ [ 0 , 1 ] .
∇ 𝑓 𝑘 ( 𝑏 𝑙 ) denotes the gradient of the classiﬁer function 𝑓 𝑘 with respect to the variable 𝑏 at 𝑏 = 𝑏 𝑙 .
Modifying from 𝑏 𝑙 to 𝑏 𝑙 ∪ 𝑙 (cid:48) or 𝑏 𝑙 ∪ 𝑗 follows the direction of gradient ascent, which gives
[Equation]
𝑓 𝑘 ( 𝑏 𝑙 ) ,𝑏 𝑙 ∪ 𝑙 (cid:48) − 𝑏 𝑙 (cid:105) ≥ {+
+} and (cid:104)∇ 𝑓 𝑘 ( 𝑏 𝑙 ) ,𝑏 𝑙 ∪ 𝑗 − 𝑏 𝑗 (cid:105) ≥ [Equation]
(cid:104)∇ ˜ [Equation] 𝑏 [Equation]
− 𝑏 𝑙
[Equation]
0 and
[Equation]
𝑓 𝑘 ( 𝑏 𝑙 ∪ 𝑙 (cid:48) )
− 𝑓 𝑘 ( 𝑏 𝑙 ) ≤
[Equation]
We can derive the lower bound of the submodularity ratio 𝛾 𝑘,𝜁 of 𝑓 𝑘 :
[Equation]
Supposing 𝜖
𝑘 < 0 for each 𝑘 ∈ { 1 ,
[Equation] [Equation]
𝐾 − 1 }
[Equation]
we can derive:
[Equation]
Given the smoothness assumption on the targeted classiﬁer (See Deﬁnition.2) [Equation] there exits a value of 𝑀 𝑘, Ω 1 | 𝜁 |/ 2 ≤ (cid:107)∇ 𝑓 𝑘 ( 𝑏 𝑙 ) 𝜁 (cid:107) 2 , which allows that (cid:107)∇ 𝑓 𝑘 ( 𝑏 𝑙 ) 𝜁 (cid:107) 2 − 𝑀 𝑘, Ω 1| 𝜁 | ≥ 0 holds.
Therefore, we can derive the lower bound of the submodularity ratio of 𝐹 𝑘 and
− 𝐺 𝐾 , 𝛾 𝑘,𝜁 (k=1,2,3,...,K):
[Equation]
[Equation]
The submodularity ratio 𝛾 𝜁 on Ω 𝜁 in Eq.26 is 𝛾 𝜁 =
[Equation]
Proof of deriving the robustness assessment score under the optimistic scenario in Eq.8.
Following the setting of Eq.26, we can formulate the robustness assessment problem at the optimistic scenario (with an oblivious adversary) :
[Equation]
Both 𝐺 𝑘 ( 𝑆 ) and ˜ 𝐹 𝑘 ( 𝑆 ) are non-increasing with ℎ 𝑘 ( 𝑏 𝑙 )
= − 𝑓 𝑘 ( 𝑏 𝑙 ) .
Eq.31 thus deﬁnes a non-monotone set function maximization problem (See Deﬁnition.3).
We conduct the analysis as in Eq.27 and Eq.29.
Modifying 𝑏 𝑙 to 𝑏 𝑙 ∪ 𝑙 (cid:48) and 𝑏 𝑙 ∪ 𝑗 follows the direction of gradient descent, which gives (cid:104)∇ 𝑓 𝐾 ( 𝑏 𝑙 ) ,𝑏 𝑙 ∪ 𝑗 − 𝑏 𝑙 (cid:105) ≤ 0 and (cid:104)∇ 𝑓 𝐾 ( 𝑏 𝑙 ) ,𝑏 𝑙 ∪ 𝑙 (cid:48) − 𝑏 𝑙 (cid:105) ≤ 0 , (cid:104)∇ ℎ 𝑘 ( 𝑏 𝑙 ) ,𝑏 𝑙 ∪ 𝑗 − 𝑏 𝑙 (cid:105) ≤ 0 and (cid:104)∇ ℎ 𝑘 ( 𝑏 𝑙 ) ,𝑏 𝑙 ∪ 𝑙 (cid:48) − 𝑏 𝑙 (cid:105) ≤ 0 .
Supposing 𝜖 ≥ 0 , we can derive:
[Equation]
From Eq.
[Equation]
we conduct the similar analysis as in Eq.22 and
[Equation]
which gives the submodularity ratio at the optimistic case:
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Let 𝛾 𝑘 ∈ ( 0 , 1 ) and − 𝑀 𝑘, Ω 1 | 𝜁 |−(cid:107)∇
[Equation]
𝑘
[Equation]
[Equation]
𝜁 |
[Equation]
|
[Equation]
−(cid:107)∇ ℎ 𝑘 ( 𝑏 𝑙 ) 𝜁 (cid:107) 2 ) . The submodularity ratio of ℎ 𝑘 , 𝛾 𝑘 ≤ 𝑚𝑘
[Equation]
Proofs to Theorem.3 (Eq.9).
In the pessimistic scenario, assessing adversarial robustness can be formulated as a problem of monotone weakly submodular function maximization.
Assuming that 𝑚 𝑓 ( 𝑙 ) is the solution 𝑙 obtained by the FSGS method and 𝑚 𝑓 ( 𝑙 ∗ ) is the optimal solution 𝑙 ∗ to Eq.4.
Based on Theorem.3 in (Elenberg et al., 2018), we can obtain:
[Equation]
Therefore, we can derive the ﬁrst inequality (the pessimistic scenario) in Eq.
[Equation]
Assessing adversarial robustness in the optimistic scenario (Eq.5) is in nature a problem of nonmonotone weakly submodular function maximization.
Assuming that 𝑚 𝑓 ( 𝑙 ) is the greedy search based solution obtained by the RandGS (Randomized Greedy Search) and 𝑚 𝑓 ( 𝑙 ∗ ) is the optimal solution to
Eq.5.
Applying Theorem 1.9 (Santiago & Yoshida, 2020) directly, we can obtain:
[Equation]
The proof to the approximation gap of OMPGS in Eq.10 follows the proof to Theorem 2.4 and Theorem 3 (Santiago & Yoshida, 2020).
D A LGORITHM PSEUDO CODES FOR FSGS AND R AND GS BASED ROBUSTNESS ASSESSMENT
←
7: 𝑀 ⊂ 𝐻 𝑙 𝑡 − 1 be a subset of size 𝜏 8: minimizing (cid:205)
𝑎 𝑖 ∈ 𝑀 𝑇 ( 𝑎 𝑖 ) ( level Optimization ) 9:
Let ˜ 𝑎 be a uniformly random element from 𝑀 10:
Let ˜ 𝑠
( ˜ 𝑎 ) be the subset corresponding to ˜ 𝑎 11: 𝑙 ← 𝑙 𝑖 − 1 ∪{ ˜ 𝑎 } 12: 𝑚 𝑓 ( 𝑙 𝑖 )← 𝑚𝑖𝑛 { 𝑚 𝑓 ( ˜ 𝑎 ∪ ˜ 𝑠 ( ˜ 𝑎 )) ,𝑚 𝑓 ( 𝑙 𝑖 − 1 )} 13: if 𝑚 𝑓 ( 𝑙 𝑖 ) ≥ Γ then break 14: end for
E O RTHOGONAL M ATCHING P URSUIT G REEDY S EARCH
The pseudo-codes of OMPGS is presented in Algorithm 3, which explains how it is adopted to solve
Eq.4 and Eq.5.
In each iteration, for each subset 𝑠 of the previous support set 𝑙 𝑘 − 1 , we compute the gradient of 𝑓 𝑦 with respect to b 𝑠 (binary indicator matrix b with the entries in 𝑠 changed).
The top- 𝑘 (cid:48) attributes with the largest magnitudes in the gradient vector are selected to form a candidate set
ˆ 𝑠 .
We can then ﬁnd out the optimal attribute 𝑗 ∈ ˆ
𝑠 to extend for each subset 𝑠 and record the optimal attack objective value 𝑔 ( 𝑙 𝑘 − 1 ∪ 𝑗 ) .
In the outer iteration, we choose ﬁnally the attribute 𝑗 ∗ producing the largest marginal gain to add into 𝑙 𝑘 − 1 .
The worst case cost of objective function evaluation in each iteration of OMPGS is bounded by 𝑂 (( (cid:205)
𝑘 − 1 𝑖 = 1 𝐶 𝑖𝑘 ))
| 𝑘 (cid:48) | .
We adjust 𝑘 (cid:48) to achieve a trade-off between enlarging the search range of greedy search and the cost of function evaluation.
Usually 𝑘 (cid:48) is much less than | 𝐻 |−| 𝑙 𝑘 | , especially when 𝐻 is signiﬁcantly large (e.g. 𝐻 > 10k).
Thus OMPGS runs signiﬁcantly faster than FSGS .
We further illustrate the rationality of using the orthogonal matching pursuit step in OMPGS to solve the weakly submodular maximization based assessment problem.
In Eq.
39, we unveil that the gradient magnitude of 𝑓 𝑦 with respect to the binary indicator variables 𝑏 𝑗𝑖 can be used as an estimator of the marginal gain of 𝑚 𝑓 in each iteration of the greedy search of OMPGS .
We assume that ˆ 𝑏 and ˆ 𝑏 (cid:48) denote two sets of categorical attribute changes.
𝑏 indicates the category value assignment of an unperturbed data instance ˆ 𝑥 .
| diff ( 𝑏, ˆ 𝑏 )| ≤ 𝜁, | diff ( 𝑏, ˆ 𝑏 (cid:48) )
| ≤ 𝜁, | diff ( ˆ 𝑏, ˆ 𝑏 (cid:48) )
| ≤ 𝜁,𝜁 ≥ 1 .
The targeted classiﬁer is denoted as 𝑓 𝑦 𝑘 ( 𝑥 ) (k=1,..., K ), in which K is the correct class label.
|
[Equation]
ˆ 𝑥,𝑏 )| ≤ 𝑚𝑎𝑥 { 2 𝑚 𝑘,𝜁 (cid:107)∇ 𝑓 𝑦 𝑘 ( 𝑥, ˆ 𝑏 ) 𝜈 (cid:107) 22 , (cid:107)∇ 𝑓
[Table]
[Table]
[Table]
𝑏 ) 𝜈 (cid:107) 2 +
[Table]
𝑘, Ω 𝜁 | 𝜁 |/ 2 } | 𝑓 𝑦 K
( ˆ 𝑥, ˆ 𝑏 (cid:48) )− 𝑓 𝑦 K ( ˆ 𝑥, ˆ 𝑏 )| ≥ 𝑚𝑖𝑛 { 2 𝑀 K ,𝜁 (cid:107)∇ 𝑓 𝑦 K ( 𝑥, ˆ 𝑏 ) 𝜈 (cid:107) 22 , (cid:107)∇ 𝑓 𝑦 K ( 𝑥, ˆ 𝑏 ) 𝜈 (cid:107) 2 + 𝑚 𝑘, Ω 𝜁 | 𝜁
|/ 2 } (39) where 𝜈 = diff ( ˆ 𝑏, ˆ 𝑏 (cid:48) ) .
F E XPERIMENTAL E VALUATION
We ﬁrst introduce the evaluation datesets in section F.1, the experimental setup in section F.2, and the complexity analysis and running time in section F.3.
Besides the multi-class datasets Yelp , IPS and Splice , we also include another two binary classiﬁcation datasets collected from Cyber Security and HealthCare in the experiments.
We add the feature sensitivity with the multi-class datasets Yelp ,
IPS and Splice in F.4 as the supplementary information to the main text.
We add the the robustness assessment with the binary-class datasets PEDec and EHR in F.5.
It includes the summarised table on the robustness assessment and the information-theoretic characterization of adversarial vulnerability veriﬁed in the experiment.
F. D ATASET INFORMATION
Yelp-5 ( Yelp )(Asghar, 2016).
We use the Yelp-5 provided in the torchtext package contains 650, training and 50,000 testing textual samples.
In the classiﬁer training process.
We learn a 300dimensional embedding for each word.
Thus, each sentence is encoded as 𝑥 ∈ R 𝑛 ∗ 300 , where 𝑛 is the number of words.
we treat every word as a categorical feature and put this data into the LSTM model with ﬁve level rating classes of the reviews as the labels.
The task is to predict the 5-class rating of each review.
After learning the classiﬁer, we start to attack the categorical input with modifying the words with those semantic neighbor words, which can contain the sentence is semantically correct.
Here not all the words have neighbors.
Here we choose the 200 data from the testing data for attack.
Intrusion Prevention System Dataset (IPS) (Wang et al., 2020).
An adversary of network intrusion performs a series of actions to compromise the targeted devices.
We collect one day of IPS records via a security service vendor from 242,467 endpoint devices.
After removing repeated instances, the collected data set contains 4,101 time series of attack events.
Each sequence instance is composed by 20 attack steps.
On each attack step, the adversary can choose one of 1,103 different malicious actions registered as highly threatening ones.
Thus one data instance of IPS data is given as x ∈ 𝑅 20 ∗ 1103 ∗according to the deﬁnition given in Section.3.
Each of the 1103 attack actions is projected to a 70dimensional embedding vector.
In our study, each sequence instance is used as input to the classiﬁcation system, which predicts the most likely attack operation conducted at the immediately successive step of the sequence.
We randomly select 80% of each dataset for training and others for testing.
Based on the prediction output, security analysts can proactively take prevention actions.
We focus on predicting the occurrence of the two most threatening actions related to recently uncovered vulnerability.
We thus study a 3-class classiﬁcation task: the 2 highly malicious actions and all the others as the third class.
All the privacy-related information, such as names of the devices, IP addresses and hosting machines’ domain names have been removed and anonymized.
Splice-junction Gene Sequences (Splice) (Noordewier et al., 1991).
Splice junctions are the points on a DNA sequence at which redundant
DNA segments are removed during the process of protein creation.
As shown in Figure 2 (Htike & Win, 2013), a precursor mRNA contains exons and introns.
During splicing, introns get spliced out while exons get reunited.
It is then important to analyze the role of splice junction sequences in mRNA.
The Splice dataset contains 3,190 samples from Genbank release 64.1.
Each of the instance is a gene fragment of 60 categorical features, which consists of a window that coversnucleotides respectively before and after each donor and acceptor site.
Each categorical feature can take one among ’A’, ’G’, ’C’, ’T’ or ’N’(stands for ambiguity).
In our work, Each category is cast to a 30-dimension embedding.
Then one data instance of the dataset is given as x ∈ 𝑅 60 ∗ 5 ∗ 30 .
The dataset deﬁnes a 3-class classiﬁcation task.
Each gene instance can be an intro-exon boundary (labelled as IE ), an exon-intro boundary (labelled as EI ), or any other cases (labelled as N ).
There are 768 IE instances and 767 EI instances in total.
All the rest are N instances.
In the published dataset, the data records are anonymized.
We randomly select 80% of each dataset for training and others for testing.
Windows PE Malware Detection (PEDec) .
We build a well-balanced dataset composed by full dynamic analysis reports of 20000 benign and malicious PE executables.
This dataset is used for a binary classiﬁcation scenario of PE malware detection.
We collect 10,972 malware samples offamilies randomly selected from those submitted to VirusTotal between 2018 and 2020.
Each of the ﬁles https://www.overleaf.com/project/5f81ef055ce9280001ecdee8is classiﬁed as malicious by more than 21 antivirus engines.
We further use AVClass to ensure that no malware families are over-represented in the collected data.
For the benign samples, we choose the fresh installation packages of all the community-maintained packages of Chocolatey, where the package submissions go through a strict review process to exclude malicious and pirated software.
Therefore we obtain 9028 benign samples of third-party and Windows system ﬁles.
Each executable of the dataset is encoded into a binary feature vectors with 5000 signatures selected by human experts.
Each signature covers presence or absence of speciﬁc windows API calls, URL access, registry table edits, ﬁle I/O operations and the status of
IP ports.
The ensemble of the signatures’ binary states form a description of the ﬁle’s behavior in execution within a sandbox.
In our study, we use the 5000 binary codes directly as the features of a ﬁle instance.
Simple as it is, the combination space of these signatures can still induce prohibitively expensive search cost, which prevent brute-force attacks.
We randomly select 80% of each dataset for training and others for testing.
Electronic Health Records (EHR) (Ma et al., 2018).
The real-world EHR dataset consists of timeordered medical visit records of 7314 patients.
Each patient has from 4 to 200 medical visits.
Each visit record is composed by a subset of 4130 discrete ICD9 diagnosis codes 2 .
Each diagnosis code represents occurrence of a disease, a symptom, or an abnormal ﬁnding.
Using the historical EHR data of patients, we can predict the risk of patients suffering the target diseases.
In this experiment, our target is a binary classiﬁcation task: we forecast whether a patient will suffer heart failure disease in the future.
In our experiments, a data instance of EHR data set is organized as a tensor x ∈ 𝑅 200 ∗ 4130 ∗ 70 with 2 http://www.icd9data.com/ each of the 4130 diagnosis codes projected to a 70-dimensional embedding vector.
For the patients with less than 200 visits, we pad the empty observations by setting the corresponding 𝑏 𝑗𝑖 = 0 .
We split randomly each dataset into two non-overlapped subsets: 80% of them are used for training and the left 20% form a testing set.
In the testing set, we choose the correctly classiﬁed testing samples with the strongest probabilistic conﬁdence provided by the classiﬁer to perform evasion attack.
There are thus 125, 500, 324, 3000 and 414 instances of Yelp , IPS , Splice , PEDec and EHR used in the evasion attack scenario.
On PEDec and EHR data, since the discrete feature takes binary values, we focus on adversarial modiﬁcation by simply ﬂipping the codes.
For Yelp , IPS and Splice , the adversarial attack is conducted by replacing the original category values of a targeted discrete attribute with a new one.
F. E XPERIMENTAL S ETUP
We instantiate the study of robustness characterization and assessment to the popularly deployed
Convolution Neural Networks (CNN) and Long Short-Term Memory (LSTM) models.
For PEDec dataset, we adopt a simple CNN model composed of one convolution layer followed by two linear layers.
The rest datasets contain sequential instances, we thus apply standard LSTM as the classiﬁer in the experiments.
Without loss of generality, we use ReLu activation function in both the CNN and LSTM classiﬁer with the dropout module.
Theorem.2 applies in both cases.
Both models deﬁne
Lipschitz-continuous decision functions, which meets the smoothness constraint given in Deﬁnition 2.
For the instances used for robustness assessment, the CNN and LSTM classiﬁer achieve accurate classiﬁcation over all the datasets.
They achieve accuracy scores of 0.90 , 0.95 , 0.94 , 0.93 and 0.
respectively for the datasets IPS , Splice , PEDec , EHR and Yelp .
In Section 4.1, we demonstrate robustness assessment of the two classiﬁers over all the datasets with varied tolerance levels against adversarial perturbation.
We set the robustness threshold Γ to − 0 .
4 andrespectively.
The former deﬁnes a low tolerance to the decision perturbation caused by adversarial attack.
A minor ﬂuctuation of the classiﬁer’s output triggers an alert of potential evasion attack.
In contrast, the latter setting only raises an alarm when the classiﬁer produces an ambiguous decision close to the boundary or report a miss-classiﬁcation incident.
Furthermore, we show the derived robustness assessment scores with respect to an oblivious adversary ( OA ,the optimistic assessment) and an knowledgeable adversary ( KA ,the pessimistic assessment).
The purpose is to show that 1) our proposed robustness assessment method can adapt varied tolerance settings to adversarial threats in safety-critic applications ; 2) we aim at conﬁrming the impact of different query capabilities of the adversary over the robustness assessment scores , as revealed by Theorem 2.
For the instances used for robustness assessment, we report the median and mean of the derived robustness assessment scores.
Both FSGS / RandGS and OMPGS are used to calculate the robustness assessment scores.
They verify the effectiveness of greedy search for robustness assessment with categorical input.
We implement the empirical study using the Python library PyTorch and conduct all the experiments on Linux server with 2 GPUs (GeForce 1080Ti) and 16-core CPU (Intel Xeon).
For RandGS and OMPGS , we empirically set the number of candidate attributes in each iteration of greedy search to be 10 globally for all the datasets.
F. T IME COMPLEXITY AND RUNNING TIME COMPARISON
We provide the time complexity analysis in Table.4.
We assume a data instance 𝑥 containing 𝑁 categorical features and each feature can choose any of the 𝑀 categorical values.
Furthermore, we assume FSGS , OMPGS and GradAttack all run 𝑇 iterations.
Table.4 shows the query complexity of these methods of 𝑇 iterations.
In the table, 𝐾 ompgs for OMPGS denotes the number of top-ranked candidate features in each iteration.
OMPGS then selects one feature from the top 𝐾 ompgs candidate features.
The selected feature achieves the best attack performance by combing this feature and any subset of the already modiﬁed features.
𝐾 denotes the number of the candidate features selected to modify in each iteration.
As seen in Table.4, FSGS has the largest query complexity in each iteration.
Enjoying the heuristics provided by the orthogonal matching pursuit step, OMPGS locates simultaneously the top-ranked candidate features and the categorical values of these candidate features that are likely to bring the most increase of 𝑚 𝑓 .
Therefore, OMPGS can reduce the query complexity signiﬁcantly compared to
FSGS .
GradAttack does not evaluate the gain of 𝑚 𝑓 on the combination of the candidate features and
[Table] [Table]
[Table]
the subsets of the already modiﬁed features.
The query complexity of GradAttack in each iteration only depends on 𝐾 grad and 𝑀 .
We also show the running-time evaluation of these methods in Table.5 and Table.6.
In both tables, we focus on the pessimistic robustness problem and set the threshold Γ= 0 in Eq.4 in the main paper.
Table.5 records the averaged running time for computing robustness assessment results over the input data instances where the evasion attack is delivered successfully,
i
[Table]
e. 𝑚 𝑓 increases above 0.
Table.
shows the average number of iterations required over the input instances to make 𝑚 𝑓 increase over 0.
As seen in the two tables, FSGS stops the loops of greedy search generally earlier than OMPGS and
GradAttack , while costing the largest running time.
The observation is consistent with the design of
FSGS .
On one hand, FSGS traverses all the candidate features and chooses the one that most improves the marginal gain.
Compared to OMPGS and GradAttack , FSGS has a better chance to achieve higher marginal gain of 𝑚 𝑓 in each iteration.
Therefore, FSGS requires fewer loops in the greedy search and modiﬁes less number of features to solve the robustness assessment problem.
However, on the other hand, FSGS makes larger query costs in each iteration of the greedy search.
The running time of FSGS is thus higher than the other two methods.
On the contrary, GradAttack runs fast but needs signiﬁcantly more iterations in the search loops.
In each iteration, GradAttack only needs to evaluate the combinations of the 𝐾 𝑔𝑟𝑎𝑑 features selected from the candidate set and choose the combination contributing the best increase of 𝑚 𝑓 .
It requires much fewer queries than FSGS , as it only searches a subset of the combinations that FSGS looks into.
However, GradAttack likely results in less optimal solutions in each iteration.
Therefore, GradAttack needs more loops to extend its search range.
OMPGS achieves the best trade-off in between.
The orthogonal matching pursuit step adopted in
OMPGS helps narrow down drastically the search range while guaranteeing the optimality of the features in the reduced search range.
Compared to FSGS , OMPGS stops early the search with a similar number of loops, yet costing much fewer queries and less running time than FSGS .
Compared to
GradAtttack , OMPGS has a better chance to select the really useful feature modiﬁcations, thus costing fewer loops.
F. F EATURE S ENSITIVITY
To further understand the association between the feature sensitivity and the derived robustness assesment scores, we conduct one-factor-at-a-time sensitivity analysis (Campbell et al., 2008) to measure the sensitivity of features.
Given a data instance, we change each feature while keeping all the others ﬁxed.
The averaged change of the classiﬁer’s output over all the testing instances is used as the feature-wise sensitivity measurement.
A larger value indicates that the classiﬁer’s output is more
[Table] [Table]
sensitive to the change over the corresponding feature.
In Table 7, we show the maximum , average and minimum decrease of the probabilistic output for IPS and Splice , induced by changing the top-3 and 5 -ranked features.
First of all, we can ﬁnd that the feature sensitivity of IPS lies in a narrow interval lower than all the other datasets.
This resonates with the larger robust assessment scores in Table 1 and 2.
Second,
Splice also has a narrow interval with high sensitivity values, indicating a potential high vulnerability.
Third, within dataset IPS and Splice , the sensitivity levels vary strongly, between the max and min value.
That implies, changing the top sensitive features only is enough to deliver a successful attack.
In real-world applications, the existence of such highly sensitive yet sparse sensitive attributes is widely witnessed.
For example in IPS , the top-5 ranked features are relevant to the key signatures of network trafﬁc ﬂows, and they are thus usually used by human security analysts to ﬁler out suspicious incidents.
The rest of the features in IPS are not explicitly relevant with the network I/O behaviours, which are less informative in the classiﬁcation task.
In Splice , the top-5 sensitive features are found as the gene segments around the spliced junctions.
They are particularly sensitive to the gene types.
The rest of the features show little physical signiﬁcance in the gene splicing process.
We also ﬁnd that the top-sensitive features also appear as the most frequently selected features by
FSGS , GradAttack and OMPGS .
We further count the frequency of appearance for each feature selected in the pessimistic robustness assessment and compare them with the top-sensitive features, as given by Table 8.
For IPS , the top-3 sensitive features also appear in the list of the most frequently selected features.
The top-4 sensitive features in Splice are also the most frequently selected 4 features.
The interesting overlapping between the attributes which are useful for attack and the top-ranked sensitive attributes conﬁrms our intuition about the association between feature sensitivity and adversarial vulnerability of the classiﬁer.
On Splice , 4 of the most frequently attacked features (29,28,30 and 31 in
Table 8) correspond to the junction of exons, which are also highly informative for gene categorization (Htike & Win, 2013).
On IPS , 3 of the 5 most frequently attacked categorical features (19, 0 and 17) are the also informative indicators of unusual DNS requests and DDos activities.
They are ranked as the most sensitive features according to Table 8.
Suppressingfeaturesensitivityhelpsimproveadversarialrobustness.
However, thesesensitiveattributes are usually informative for classiﬁcation in an adversary-free scenario.
Excluding the sensitive features causes accuracy loss.
It indicates the balance between classiﬁcation utility and adversary-resilience feature engineering in ML practices.
F. R OBUSTNESS A SSESSMENT IN B INARY C LASSIFICATION F.5. R OBUSTNESS ASSESSMENT WITH GREEDY SEARCH
Table 9 shows the results of robustness assessment for models built on binary-class datasets PEDec and EHR .
We have the same ﬁndings in that with the multi-class datasets Yelp , IPS and Splice .
In details, ﬁrstly for both the pessimistic and optimistic robustness assessment problems, a lower tolerance threshold Γ results in lower scores over all the binary-class datasets.
Secondly the pessimistic robustness assessment scores are always lower or equal to the optimistic assessment scores.
F.5. R OBUSTNESS VS F EATURE I NFORMATIVENESS
We take the 𝑥 𝑙 𝑚𝑖 samples from the 62, 43 original samples for the dataset PEDec , EHR respectively on the half-way of pushing the 𝑚 𝑓 of the original samples ( OS ) to surpass Γ .
In Table 10, we veriﬁed
[Table] [Table]
[Table]
the impact of feature informativeness on the robustness assessment scores at Γ = 0 with the binaryclass datasets PEDec and EHR .
The table shows that the classiﬁer becomes less robust to adversarial modiﬁcations over 𝑥 𝑙 𝑚𝑖 with less predicative information.
F.5. R OBUSTNESS A SSESSMENT AFTER R OBUSTNESS E NHANCEMENT
From the parameter of the decreased SR in Table 11, we study the robustness enhancement by different model enhancement techniques.
On PEDec , the most robustness assessment scores increase slightly.
Notably, the decreased SR of AdvC and NuR are positive and brings the most robustness improvement.
On EHR , NuR can prevent attack over all the testing instances.
As seen in Table 11, the robustnessenhancing techniques induce more visible impact on the pessimistic robustness assessment scores using OMPGS .
This result meets the intuitive interpretations in the main paper.