S OUND AND C OMPLETE N EURAL N ETWORK R EPAIR WITH M INIMALITY AND L OCALITY G UARANTEES
We present a novel methodology for repairing neural networks that use ReLU activation functions.
Unlike existing methods that rely on modifying the weights of a neural network which can induce a global change in the function space, our approach applies only a localized change in the function space while still guaranteeing the removal of the buggy behavior.
By leveraging the piecewise linear nature of ReLU networks, our approach can efﬁciently construct a patch network tailored to the linear region where the buggy input resides, which when combined with the original network, provably corrects the behavior on the buggy input.
Our method is both sound and complete – the repaired network is guaranteed to ﬁx the buggy input, and a patch is guaranteed to be found for any buggy input.
Moreover, our approach preserves the continuous piecewise linear nature of ReLU networks, automatically generalizes the repair to all the points including other undetected buggy inputs inside the repair region, is minimal in terms of changes in the function space, and guarantees that outputs on inputs away from the repair region are unaltered.
On several benchmarks, we show that our approach signiﬁcantly outperforms existing methods in terms of locality and limiting negative side effects.
I NTRODUCTION
Deep neural networks (DNNs) have demonstrated impressive performances on a wide variety of applications ranging from transportation Bojarski et al.
(2016) to health care Shahid et al.
However, DNNs are not perfect.
In many cases, especially when the DNNs are used in safety-critical contexts, it is important to correct erroneous outputs of a DNN as they are discovered after training.
For instance, a neural network in charge of giving control advisories to the pilots in an aircraft collision avoidance system, such as the ACAS Xu network from Julian et al.
(2019), may produce an incorrect advisory for certain situations and cause the aircraft to turn towards the incoming aircraft, thereby jeopardizing the safety of both airplanes.
In this paper, we consider the problem of neural network repair , i.e. given a trained neural network and a set of buggy inputs (inputs on which the neural network produces incorrect predictions), repair the network so that the resulting network on those buggy inputs behave according to some given correctness speciﬁcation.
Ideally, the changes to the neural network function should be small so that the outputs on other inputs are either unchanged or altered in a small way.
Existing efforts on neural network repair roughly fall into the following three categories.
Retraining/ﬁne-tuning.
The idea is to retrain or ﬁne-tune the network with the newly identiﬁed buggy inputs and the corresponding corrected outputs.
Methods include counterexample-guided data augmentation Dreossi et al.
; Ren et al.
(2020), editable training Sinitsin et al.
and
training input selection Ma et al.
One major weakness of these approaches is the lack of formal guarantees – at the end of retraining/ﬁne-tuning, there is no guarantee that the given buggy inputs are ﬁxed and no new bugs are introduced.
In addition, retraining can be very expensive and requires access to the original training data which is impractical in cases where the neural network is obtained from a third party or the training data is private.
Fine-tuning, on the other hand, often faces the issue of catastrophic forgetting Kirkpatrick et al. [Figure]
𝑥
Direct weight modiﬁcation.
These approaches directly manipulate the weights in a neural network to ﬁx the buggy inputs.
The repair problem is typically cast into an optimization problem or a veriﬁcation problem.
For example, Dong et al.
(2020) proposes to minimize a loss deﬁned on the buggy inputs.
Goldberger et al.
(2020) uses an SMT solver to identify minimal weight changes to the output layer of the network so that the undesirable behaviors are removed.
In general, the optimization-based approach cannot guarantee removal of the buggy behaviors, and the veriﬁcationbased approach does not scale beyond networks of a few hundred neurons.
In addition, both approaches suffer from substantial accuracy drops on normal inputs since weight changes may be a poor proxy for changes in the function space .
Architecture extension.
The third category of approaches extends the given NN architecture, such as by introducing more weight parameters, to facilitate more efﬁcient repairs.
The so-called
Decoupled DNN architecture Sotoudeh & Thakur (2021) is the only work we know that falls into this category.
Their idea is to decouple the activations of the network from values of the network by augmenting the original network.
Their construction allows the formulation of any single-layer repair as an linear programming (LP) problem.
The decoupling, however, causes the repaired network to become discontinuous (in the functional sense).
In addition, it still cannot isolate the output change to a single buggy input from the rest of the inputs.
In addition to the aforementioned limitations, a common weakness that is shared amongst these methods is that the induced changes, as a result of either retraining or direct weight modiﬁcation, are global .
This means that a correct behavior on another input, regardless of how far it is away from the buggy input, may not be preserved by the repair.
Worse still, the repair on a new buggy input can end up invalidating the repair on a previous buggy input.
The fundamental issue here is that limiting the changes to a few weights or a single layer only poses a structural constraint (often for ease of computation); it does not limit the changes on the input-output mapping of the neural network.
It is known that even a single weight change can have a global effect on the output of a neural network.
In this paper, we propose REASSURE , a novel methodology for neural network repair with locality, minimality, soundness and completeness guarantees.
Our methodology targets continuous piecewise linear (CPWL) neural networks, speciﬁcally those that use the ReLU activation functions.
The key idea of our approach is to leverage the CPWL property of ReLU networks to synthesize a patch network tailored to the linear region where the buggy input resides, which when combined with the original network, provably corrects the behavior on the buggy input .
Our approach is both sound and complete – the repaired network is guaranteed to ﬁx the buggy input, and a patch is guaranteed to be found for any buggy input.
Moreover, our approach preserves the CPWL nature of ReLU networks, automatically generalizes the repair to all the points including other undetected buggy inputs inside the repair region, is minimal in terms of changes in the function space, and guarantees that outputs on inputs away from the repair region are unaltered.
Figure 1 provides an illustrative comparison of our approach with other methods.
Table 1 compares our approach with representative related works in terms of theoretical guarantees.
We summarize our contributions below.
• We present REASSURE , the ﬁrst sound and complete repair methodology for ReLU networks with strong theoretical guarantees.
• Our technique synthesizes a patch network, which when combined with the original neural network, provably corrects the behavior on the buggy input.
This approach is a signiﬁcant departure from existing methods that rely on retraining or direct weight manipulation.
[Table] • Our technique is both effective and efﬁcient – across a set of benchmarks, REASSURE can efﬁciently correct a set of buggy inputs or buggy areas with little or no change to the accuracy and overall functionality of the network.
B ACKGROUND 2. D EEP N EURAL N ETWORKS
An R -layer feed-forward DNN f = κ R ◦ σ ◦ κ R − 1 ◦ ... ◦ σ ◦ κ 1 : X → Y is a composition of linear functions κ r , r = 1 , 2 , ..., R and activation function σ , where
X ⊆
R m is a bounded input domain and
Y ⊆ R n is the output domain.
Weights and biases of linear function { κ r } r =1 , 2 ,...,R are parameters of the DNN.
We call the ﬁrst R − 1 layers hidden layers and the R -th layer the output layer.
We use z ij to denote the i -th neuron (before activation) in the j -th hidden layer.
In this paper, we focus on ReLU DNNs, i.e. DNNs that use only the ReLU activation functions.
It is known that an R m → R function is representable by a ReLU DNN if and only if it is a continuous piecewise linear (CPWL) function Arora et al. (2016).
The ReLU function is deﬁned as σ ( x ) = max ( x, 0) .
We say that σ ( x ) is activated if σ ( x ) = x .
L INEAR R EGIONS
A linear region A is the set of inputs that correspond to the same activation pattern in a ReLU DNN f Serra et al.
Geometrically, this corresponds to a convex polytope, which is an intersection of half spaces, in the input space X on which f is linear.
We use f | A to denote the part of f on A .
2. C ORRECTNESS S PECIFICATION
A correctness speciﬁcation Φ =
(Φ in , Φ out ) is a tuple of two polytopes, where Φ in is the union of some linear regions and Φ out is a convex polytope.
A DNN f is said to meet a speciﬁcation Φ =
(Φ in , Φ out ) , denoted as f | =
Φ , if and only if ∀ x ∈ Φ in , f ( x ) ∈ Φ out .
Example 1.
For a classiﬁcation problem, we can formally write the speciﬁcation that “the prediction of any point in an area A is class k ” as Φ =
(Φ in , Φ out ) , where Φ in = A and Φ out = { y ∈ R n | y k ≥ y i , ∀ i (cid:54) =
k } 1 .
2. P ROBLEM D EFINITION
In this paper, we consider the following two repair problems.
Deﬁnition 1 (Area repair) .
Given a correctness speciﬁcation Φ =
(Φ in , Φ out ) and a ReLU DNN f
(cid:54)| =
Φ , the area repair problem is to ﬁnd a modiﬁed ReLU DNN (cid:98) f such that (cid:98) f | = Φ .
Note that we do not require (cid:98) f to have the same structure or parameters as f in this deﬁnition.
If Φ in contains a single (buggy) linear region, we refer to this as single-region repair .
If Φ in contains multiple (buggy) linear regions, we refer to it as multi-region repair .
Deﬁnition 2 (Point-wise repair) .
Given a set of buggy inputs { (cid:101) x 1 , . . .
, (cid:101) x L } ⊂
Φ in with their corresponding correct outputs { y 1 , . . .
, y L } and a ReLU DNN f , the point-wise repair problem is to ﬁnd a modiﬁed ReLU DNN (cid:98) f such that ∀ i, (cid:98) f
( (cid:101) x i ) =
y i .
We call the minimal variants of area repair and point-wise repair minimal area repair and minimal point-wise repair respectively.
Minimality here is deﬁned with respect to the maximum distance between f and (cid:98) f over the whole input domain X .
A point-wise repair can be generalized to an area repair through the following result.
2. F ROM B UGGY I NPUTS TO B UGGY L INEAR R EGIONS
The linear region where an input x resides can be computed as follows.
Lemma 1.
Lee et al.
(2019) Consider a ReLU DNN f and an input x ∈ X .
For every neuron z ij , it induces a feasible set
[Equation]
The set A ( x ) =
∩ i,j A ij ( x ) is the linear region that includes x .
Note that A ( x ) is essentially the
H-representation of the corresponding convex polytope.
2. R EPAIR D ESIDERATA
We argue that an effective repair algorithm for ReLU DNN should satisfy the following criteria.
Preservation of CPWL :
Given that the original network f models a CPWL function, the repaired network (cid:98) f should still model a CPWL function.
Soundness : A sound repair should completely remove the known buggy behaviors, i.e. it is a solution to the point-wise repair problem deﬁned in Deﬁnition 2. 3.
Completeness : Ideally, the algorithm should always be able ﬁnd a repair for any given buggy input if it exists.
Generalization : If there exists another buggy input (cid:101) x (cid:48) in the neighborhood of (cid:101)
x (e.g. the same linear region), then the repair should also ﬁx it.
For example, suppose we have an (cid:101) x
that violates a speciﬁcation which requires the output to be within some range.
It is almost guaranteed that there exists another (and inﬁnitely many) (cid:101) x (cid:48) in the same linear region that also violates the speciﬁcation.
Locality : We argue that a good repair should only induce a localized change to f in the function space.
For example, in the context of ReLU DNN, if a linear region B does not border the repair region A , i.e. B∩A = ∅ , then (cid:98) f | B ( x ) = f | B ( x ) .
Minimality : Some notion of distance between f and (cid:98) f such as max | f − (cid:98) f
| should be minimized.
Note that this is a signiﬁcant departure from existing methods that focus on minimizing the change in weights which has no guarantee on the amount of change in the function space.
Limited side effect : Repairing a buggy point should not adversely affect points that were originally correct.
For example, repairing a buggy input (cid:101) x in region A should not change another region from correct to incorrect.
Formally, for any linear region C who is a neighbor of A , i.e. C ∩ A (cid:54) = ∅ , if f | C | =
Φ , then (cid:98) f | C | = Φ . 8.
Efﬁciency : The repair algorithm should terminate in polynomial time with respect to the size of the neural network and the number of buggy inputs.
O UR A PPROACH
We will ﬁrst describe our approach to single-region repair and then present our approach to multiregion repair which builds on results obtained from the single-region case.
Given a linear region A , our overarching approach is to synthesize a patch network h
A such that (cid:98) f
= f + h A and (cid:98) f | = Φ .
The patch network h
A is a combination of two sub-networks: a support network g A , which behaves like a characteristic function, to determine whether an input is in A , and an afﬁne patch function network p A ( x ) =
c x + d to ensure ( f + p A ) | =
Φ on A .
𝑥 !
𝑥 " 𝑧 !
𝑧 " 𝑦 −     𝑥 !
𝑥 " 𝑧 !
𝑧 " 𝑦 𝑝 ! 𝑔 ! Support Network 𝑔 ! ℎ % 3. R UNNING E XAMPLE
We use the following example to illustrate our idea.
Example 2.
Consider repairing the ReLU DNN f in Figure 2 according to the correctness speciﬁcation Φ : ∀ x ∈ [0 , 1] 2 , y ∈ [0 , 2] .
The DNN consists of a single hidden layer with two neurons zand z 2 , where
y = σ ( z 1 ) + σ ( z 2 ) , z 1 = x 1 + 2 x 2 − 1 and z 2 = 2 x 1 − x 2 .
The only linear region that violates our speciﬁcation is A = { x | 1 ≥ x 1 , 1 ≥ x 2 , x 1 + 2 x 2 − 1 ≥ 0 , 2 x 1 − x 2 ≥ 0 } .
( (cid:101) x
= (0 . 9 , 0 . 9) ∈ [0 , 1] 2 but f ( (cid:101) x )
= 2 .
6 / ∈ [0 , 2] )
The network f ( x ) on the linear region A is the afﬁne function f | A ( x ) = 3 x 1 + x 2 − 1 .
Our algorithm ﬁrst sets up an afﬁne function
p A ( x ) that minimally repairs f on A , such that ∀ x ∈
A , f ( x ) + p A ( x ) ∈ [0 , 2] .
Later in the paper, we will show p A ( x ) can be found by solving a LP problem.
The resulting patch function is p A ( x )
= − 12 x 1 − 12 x 2 .
However, directly apply f ( x ) + p A ( x ) as the patch network will have side effects on areas outside of A .
Our strategy is to combine p A ( x ) with a support network g A ( x ) which outputs 1 on A and drops to 0 quickly outside of A .
The ﬁnal repaired network is f ( x ) + σ ( p A ( x ) + g A ( x, 10) − 1) − σ ( − p A ( x ) + g A ( x, 10) − 1) .
This structure makes p A almost only active on A and achieve a localized repair.
Observe that this is still a ReLU DNN.
3. S UPPORT N ETWORKS
Support networks are neural networks with a special structure that can approximate the characteristic function of a convex polytope.
They are keys to ensuring localized repairs in our algorithm.
Assume that the linear region we need to repair is A = { x | a i x ≤ b i , i ∈ I } , where | I | is the number of linear inequalities.
The support network of A is deﬁned as:
[Equation]
where g ( x, γ ) = σ ( γx + 1) − σ ( γx ) and γ ∈ R is a parameter of our algorithm that controls how quickly g A ( x, γ ) goes to zero outside of A .
Remark :
For any x ∈ A , we have g A ( x, γ ) = 1 , i.e. the support network is fully activated.
For any x / ∈ A , if for one of i ∈ I , we have a i x − b i ≤ − 1 /γ , then g A ( x, γ ) = 0 .
Observe that g A ( x, γ ) is not zero when x is very close to A due to the requirement for preserving
CPWL.
In Theorem 3, we prove that we can still guarantee limited side effects on the whole input domain outside of A with this construction.
3. A FFINE P ATCH F UNCTIONS
We consider an afﬁne patch function p A ( x ) =
c x + d , where matrix c and vector d are undetermined coefﬁcients.
In a later section, the design of patch network will ensure that on the patch area A , the repaired network is f ( x ) + p A ( x ) .
We will ﬁrst consider ﬁnding appropriate c and d such that f ( x ) + p A ( x ) satisfy the speciﬁcation on A .
To satisfy the speciﬁcation Φ , we need f ( x ) + p A ( x ) ∈ Φ out for all x ∈ A .
To obtain a minimal repair, we minimize max x ∈A | p A ( x ) | .
Thus, we can formulate the following optimization problem
[Equation]
Notice that this is not a LP since both c and x are variables and we have a c x term in the objective.
In general, one can solve it by enumerating all the vertices of A .
Suppose that { v s | s = 1 , 2 , ..., S } is the set of vertices of A .
Since Φ out is a convex polytope, we have
[Equation]
and
[Equation]
Hence, we can solve the following equivalent LP.
[Equation]
where H ∈ R and will take max s =1 , 2 ,...,S | cv s + d | when optimal.
In general, the number of vertices of a convex polytope can be exponential in the size of its Hrepresentation Henk et al.
and enumerating the vertices of a convex polytope is known to be expensive especially when the input dimension is large Bremner (1997).
In Appendix 7.1, we show that we can solve programming 3 via LP without vertex enumeration for many useful cases such as the classiﬁcation problem in Example 1 and make our algorithm much more efﬁcient.
3. S INGLE -R EGION R EPAIRS
With a support network g A and an afﬁne patch function p A , we can synthesize the ﬁnal patch network as follows:
[Equation]
where K is a vector where every entry is equal to the upper bound of {| p A ( x ) | + ∞ | x ∈ X } .
Remark:
For x ∈ A , g A ( x, γ ) = 1 , then we have h A ( x, γ )
=
σ ( p A ( x )) − σ ( − p A ( x )) =
p A ( x ) .
For x / ∈ A , g A ( x, γ ) goes to zero quickly if γ is large.
When g A ( x, γ ) = 0 , we have h A ( x, γ )
=
σ ( p A ( x ) − K ) − σ ( − p A ( x ) − K ) = 0 .
The repaired network (cid:98) f ( x ) = f
( x ) + h A ( x, γ ) .
Since f and h A are both ReLU DNNs, we have (cid:98) f is also a ReLU DNN.
We will give the formal guarantees on correctness in Theorem 1.
3. M ULTI -R EGION R EPAIRS
Suppose there are two linear regions, A 1 and A 2 , that need to be repaired, and we have generated the afﬁne patch function p
A 1 ( x ) for A 1 and p A 2 ( x ) for A 2 .
If A 1 ∩ A 2 = ∅ , then we can repair f ( x ) with (cid:98) f
( x ) =
f
( x ) + h
A 1 ( x, γ )
+
h
A 2 ( x, γ ) directly, since h A 1 ( x, γ ) and h A 2 ( x, γ ) will not be nonzero at the same time when γ is large enough.
However, if A 1 ∩ A 2 (cid:54) = ∅ , for any x ∈ A 1 ∩ A 2 , both h A 1 ( x, γ ) and h A 2 ( x, γ ) will alter the value of f on x , which will invalidate both repairs and cannot guarantee that the repaired DNN will meet the speciﬁcation Φ .
To avoid such over-repairs, our strategy is to ﬁrst repair A 1 ∪ A 2 with p A 1 ( x ) , and then repair A 2 with p A 2 ( x ) − p A 1 ( x ) .
Figure 3 provides an illustration of a three-region case. In general, for multi-region repair, we note {A l } l =1 , 2 ,...,L are all the buggy linear regions.
Then we compute the support network g A l ( x, γ ) and afﬁne patch function p A l ( x ) for each A l . Note that this computation can be done in parallel.
A 1 : f
A 2 : f A 3 : f A 3 : f + p A⇒
A 1 : f + p A
A 2 : f + p A
A 3 : f + p A⇒
A 1 : f + p A
A 2 : f + p A
A 3 : f + p A⇒
A 1 : f + p A
A 2 : f + p A
A 3 : f + p A
Once we have g A l ( x, γ ) and p A l ( x ) , we can “stitch” multiple local patches into a ﬁnal patch as follows.
[Equation]
where K l is the upper bound of {| p
A l ( x ) − p
[Equation]
∞ | x ∈ X } and p A 0 ( x ) = 0 .
Remark : max j ≥ l { g A j ( x, γ ) } is a support function for ∪
j ≥
l
A j and its value is 1 for any x ∈ ∪ j ≥
l A j .
T HEORETICAL G UARANTEES
In this section, we present the theoretical guarantees that REASSURE provides, and point the readers to proofs of the theorems in the Appendix.
Theorem 1 (Soundness) .
The repaired DNN (cid:98) f returned by REASSURE is guaranteed to satisfy the speciﬁcation Φ .
Theorem 2 (Completeness) .
REASSURE can always ﬁnd a solution to the minimal point-wise repair or the minimal area repair problem.
For any A , the support network ensures that the patch network goes to zero quickly when x is away from A .
However, it still makes a small change on the neighbors of A .
The following theorem shows that for a big enough γ , the patch network would not change a correct region into incorrect.
Theorem 3 (Limited Side Effect) .
Given a correctness property Φ =
(Φ in , Φ out ) , a patch region
A and the corresponding patch network h ( x, γ ) , there exists a positive number Γ such that for any γ ≥ Γ , we have
for any linear region B , if B ∩ A = ∅ , then (cid:98) f ( x, γ ) =
f ( x ) ; 2.
for any linear region C who is a neighbor of A ( C∩A (cid:54) = ∅ ), if f | C | = Φ , then (cid:98) f C ( x, γ ) | = Φ .
Corollary 1 (Incremental Repair) .
For multiple-region repair, the patch for a new region A (cid:48) would not cause a previous patched region A to become incorrect.
Theorem 4 (Minimum Repair) .
For any ReLU DNN ˜ f , which is linear on a patch region A and satisﬁes the speciﬁcation Φ , there exists a positive number Γ , such that for all γ ≥ Γ ,
[Equation]
Theorem 5 (Polynomial-Time Efﬁciency) .
REASSURE terminates in polynomial-time in the size of the neural network and the number of buggy linear regions when Φ out takes the form of { y | q l ≤
Py ≤ q u } where P is a full row rank matrix and −∞ ≤ q l [ i ] ≤ q u [ i ] ≤ + ∞ ( q l [ i ] and q u [ i ] are the i -th elements of q l and q u respectively).
E XPERIMENTS
In this Section, we compare REASSURE with state-of-the-art methods on both point-wise repairs and area repairs .
The experiments were designed to answer the following questions: (Effectiveness)
[Table]
How effective is a repair in removing known buggy behaviors?
(Locality)
How much side effect (i.e. modiﬁcation outside the patch area in the function space) does a repair produce?
(Function Change)
How much does a repair change the original neural network in the function space?
(Performance)
Whether and how much does a repair adversely affect the overall performance of the neural network?
We consider the following evaluation criteria :
Efﬁcacy (E) : % of given buggy points or buggy linear regions that are repaired.
Norm Difference (ND) : average normalized norm ( L ∞ or L 2 ) difference between the original DNN and the repaired DNN on a set of inputs (e
g. training and testing data; more details in the tables).
We use ND to measure how a repair change the original neural network on function space .
Norm Difference on Patch Area (NDP) : average normalized norm ( L ∞ or L 2 ) difference between the original DNN and the repaired DNN on patch areas (calculated on random sampled points on patch areas or near the buggy points; details in the tables).
We use NDP to measure the locality of a repair.
Accuracy (Acc) : accuracy on training or testing data to measure the extent to which a repair preserves the performance of the original neural network.
Negative Side Effect (NSE) : NSE is only for area repair.
It is the percentage of correct linear regions (outside of patch area) that become incorrect after a repair.
If a repair has a nonzero NSE, the new repair may invalidate a previous repair and lead to a circular repair problem.
All experiments were run on an Intel Core i5 @ 3.4 GHz with 32 GB of memory.
We use Gurobi
Gurobi Optimization, LLC (2021) to solve the linear programs.
We compared REASSURE with the representative related works in Table 1.
REASSURE , MDNN and
PRDNN guarantee to repair all the buggy points (linear regions).
Retrain and Fine-Tuning cannot guarantee 100% efﬁcacy in general and we run them until all the buggy points are repaired.
5. P OINT - WISE R EPAIRS : MNIST
We train a ReLU DNN on the MNIST dataset LeCun (1998) as the target DNN.
The goal of a repair is to ﬁx the behaviors of the target DNN on buggy inputs that are found in the test dataset.
Thus, the repaired DNN is expected to produce correct predictions for all the buggy inputs.
The results are shown in Table 2.
REASSURE achieves almost zero modiﬁcation outside the patch area (ND) amongst all four methods.
In addition, REASSURE produces the smallest modiﬁcation on the patch area (NDP) and preserves the performance of the original DNN (almost no drop on Acc).
[Table]
Results on PRDNN using the (which are inferior to using the ﬁrst ) are shown in
Table 6 in the Appendix 7.4.
The the original
5. A REA R EPAIRS : HCAS
To the best of our knowledge
Sotoudeh & Thakur (2021) is the only other method that supports area repairs.
In this experiment we compare REASSURE with Sotoudeh & Thakur (2021) on an experiment where the setting is similar to the 2D Polytope ACAS Xu repair in their paper.
Sotoudeh & Thakur (2021) does not include a vertex enumeration tool (which is required for setting up their LP problem) in their code.
We use pycddlib Troffaes (2018) to perform the vertex enumeration step when evaluating PRDNN.
Note that the vertex enumeration tool does not affect the experimental results except running time.
We consider an area repair where the target DNN is the HCAS network (simpliﬁed version of ACAS
Xu) 2 N 1 , 4 (previous advisory equal to 1 and time to loss of vertical separation equal to 20 s ) from
Julian & Kochenderfer (2019).
We use Speciﬁcation 1 (details in Appendix 7.4), which is similar to Property 5 in Katz et al.
We compute all the linear regions for N 1 , 4 in the area Φ in of
Speciﬁcation 1 and 87 buggy linear regions were found.
We apply both REASSURE and PRDNN to repair those buggy linear regions.
We use Speciﬁcation 2 (details in Appendix 7.4), the dual of
Speciﬁcation 1, to test the negative side effect (NSE) of a repair.
The results are shown in Table 3.
Both REASSURE and PRDNN successfully repair all the buggy linear regions.
REASSURE produces repairs that are signiﬁcantly better in terms of locality (ND), minimality (NDP) and performance preservation (Acc).
5. F EATURE -S PACE R EPAIRS
In general, when repairing a large DNN with a high input dimension, the number of linear constraints for one patch area A will be huge and pose a challenge to solving the resulting LP.
One advantage of our approach, which can be used to mitigate this problem, is that it allows for point-wise and area repairs in the feature space in a principled manner, i.e. constructing a patch network starting from an intermediate layer.
This approach still preserves soundness and completeness, and is fundamentally different from just picking a single layer for repair in PRDNN or MDNN.
Experimental results on repairing AlexNet Krizhevsky et al.
(2012) for the ImageNet dataset Russakovsky et al.
(2015) in Appendix 7.4 show REASSURE (feature space) is still signiﬁcantly better in term of locality (ND) and minimality (NDP).
C ONCLUSION
We have presented a novel approach for repairing ReLU DNNs with strong theoretical guarantees.
Across a set of benchmarks, our approach signiﬁcantly outperforms existing methods in terms of efﬁcacy, locality, and limiting negative side effects.
Future directions include further investigation on feature-space repairs and identifying a lower-bound for γ .
A PPENDIX 7. R EPAIR VIA L INEAR P ROGRAMMING
We consider the case Φ out can be expressed as { y | q l ≤ Py ≤ q u } where P is a full row rank matrix and −∞ ≤ q l [ i ] ≤ q u [ i ] ≤ + ∞ ( q l [ i ] and q u [ i ] are the i -th elements of q l and q u respectively).
Consider the following optimization problem.
[Equation]
where T : R n → R n is a linear transformation on the DNN’s output space R n .
Theorem 6.
On linear region A , we have f | A ( x ) =
f 1 x + f 2 for some matrix f 1 and vector f 2 .
Assuming that f 1 is full rank 3 , the optimization problem in (3) and the optimization problem in (10) are equivalent.
Note that q l ≤ P ( T ( f ( x ))) ≤ q u can be achieved row by row.
Thus, we can ﬁnd a one-dimensional linear transformation via LPs and combine them into a single linear transformation to solve optimization problem 10.
For every row of P , say the i -th row, we can check the lower bound and upper bound of { P ( f ( x ))
| ∀ x ∈ A} on the i -th dimension by solving the following LP problems
[Equation]
(11) where P [ i ] is the i -th row of P .
Then for each row i , we take a minimal linear transformation V
[ i ]( x ) = v 1 [ i ]( x ) + v 2 [ i ] to transfer interval [ lb [ i ] , ub [ i ]] inside interval [ q l [ i ] , q u [ i ]] .
We can take v 1 [ i ] = 1 if q u [ i ] − q l [ i ] > ub [ i ] − lb
[ i ] , else v 1 [ i ] =
q u [ i ] − q l [ i ] ub [ i ] − lb [ i ] .
And v 2 [ i ] =
q l [ i ] − v 1 [ i ] lb
[ i ] if | q l [ i ] − v 1 [ i ] lb
[ i ] | ≤ | v 1 [ i ] ub [ i ] − q u
[ i ] | , else v 1 [ i ] ub [ i ] − q u [ i ] .
Since matrix P is full row rank, we can ﬁnd a linear transformation T that is equivalent to V :
[Equation]
where ˆ P = (cid:20) PP ⊥ (cid:21) is an orthogonal extension of P ( P and P ⊥ are orthogonal to each other and ˆ P is a full rank square matrix).
Once we have T , we can obtain an afﬁne patch function p A ( x )
=
T ( f ( x )) − f ( x ) .
7. T HE REASSURE A LGORITHM Algorithm 1 REASSURE
rank.
7. P ROOFS OF T HEOREMS
We prove Theorem 1 after Corollary 1, since the proof of Theorem 1 uses the result of Corollary 1.
Theorem 6.
On linear region A , we have f | A ( x ) =
f 1 x + f 2 for some matrix f 1 and vector f 2 .
Assuming that f 1 is full rank 4 , the optimization problem in (3) and the optimization problem in (10) are equivalent.
Proof.
On one side, for any c , d , since f 1 is full rank, there exists a linear transformation T , such that T ( f ( x ))
=
T ( f 1 x + f 2 ) =
( f 1 + c ) x
+ ( f 2 + d ) =
f
( x ) + c x + d .
On the other side, for any T , since T ( f ( x )) − f ( x ) is linear, there exist c , d , such that c x + d =
T ( f ( x )) − f ( x ) .
Lemma 2.
The repaired DNN (cid:98) f returned by REASSURE is guaranteed to satisfy the speciﬁcation Φ on patch area
A in single-region repair case.
Proof.
By the deﬁnition of p A , we have f ( x ) + p A ( x ) ∈ Φ out for all x ∈ A .
For any x ∈ A , we have g A ( x, γ )
= 1 and h A ( x, γ )
=
σ ( p A ( x )) − σ ( − p A ( x )) =
p A ( x ) .
Therefore,
[Equation]
Thus, the patched neural network (cid:98) f meets the speciﬁcation Φ on A .
Theorem 2 (Completeness) .
REASSURE can always ﬁnd a solution to the minimal point-wise repair or the minimal area repair problem.
Proof.
For every patch area A , we can always ﬁnd a support network g A .
For any Φ out
and A , there exists an afﬁne function p
A such that p A ( x ) ∈ Φ out , ∀ x ∈ A .
Therefore, the LP (6) is always feasible and REASSURE can ﬁnd an afﬁne patch function p A .
Once we have g A and p A for patch area A , REASSURE returns a patch network either by Equation (7) or by Equation (8).
Theorem 3 (Limited Side Effect) .
Given a correctness property Φ =
(Φ in , Φ out ) , a patch region
A and the corresponding patch network h ( x, γ ) , there exists a positive number Γ such that for any γ ≥ Γ , we have
for any linear region B , if B ∩ A = ∅ , then (cid:98) f ( x, γ ) =
f ( x ) ; 2.
for any linear region C who is a neighbor of A ( C∩A (cid:54) = ∅ ), if f | C | = Φ , then (cid:98) f C ( x, γ ) | = Φ .
Proof.
Since a multi-region repair is a composition of multiple singe-region repairs according to
Equation (8), we can prove the limited side effect of a multi-region repair by proving the limited side effect of its constituent singe-region repairs.
Below, we prove the limited side effect of a singeregion repair.
Consider patch area
A = { x | a i x ≤ b i , i ∈ I } and A > 0 ( γ ) =
{ x | h ( x, γ ) > 0 } .
Since the number of neighbors for A are ﬁnite, we can take a big enough γ , such that for any B , if B ∩ A = ∅ , B ∩ A > 0 ( γ ) = ∅ .
Thus, we have (cid:98) f
( x, γ ) =
f ( x ) on B . 2.
For any linear region C who is a neighbor of A , i.e. C (cid:54) =
A and C ∩ A (cid:54) = ∅ , (cid:98) f is no longer a linear function on C , since there are some hyperplanes introduced by our repair that will divide C into multiple linear regions.
Speciﬁcally, those hyperplanes are { x | γ ( a i x − b i ) + 1 = 0 } for i ∈
I , { x | (cid:80) i ∈
I g ( a i x − b i , γ ) − | I | + 1 = 0 } , { x | p ( x ) + K · g A ( x, γ ) − K = 0 } and { x | − p ( x ) + K · g A ( x, γ ) − K
= 0 } .
For any point x in those hyperplanes, it will fall into one of the following four cases
(a) x ∈ { x | γ ( a i x − b i ) + 1 = 0 } for some i ∈
I , then g A ( x, γ ) = 0 , h ( x, γ )
=
and (cid:98) f ( x ) ∈
Φ out ; (b) x ∈ { x | (cid:80) i ∈
I g ( a i x − b i , γ ) − | I | + 1 = 0 } , then g A ( x, γ ) = 0 ,
h ( x, γ )
=
and (cid:98) f ( x ) ∈
Φ out ; (c) x ∈ { x | p ( x ) + K · g A ( x, γ ) − K
=
0 } , then p ( x ) =
K − K · g A ( x, γ ) ≥ 0 , − p ( x ) + K · g A ( x, γ ) − K ≤ 0 , h ( x, γ ) = 0 and (cid:98) f
( x ) ∈ Φ out ; (d) x ∈ { x | − p ( x ) + K · g A ( x, γ ) − K } , then p ( x )
= K · g A ( x, γ ) − K ≤ 0 , p ( x ) + K · g A ( x, γ ) − K ≤ 0 , h ( x, γ ) = 0 and (cid:98) f
( x ) ∈
Φ out ;
By the above analysis, we have (cid:98) f ( x ) ∈ Φ out for the boundary of the new linear regions.
Since (cid:98) f is linear on the new linear regions and Φ out is convex, (cid:98)
f ( x ) ∈
Φ out for any x ∈ C .
Remark :
By Theorem 3, we have that a patch would not change a correct linear region to an incorrect one.
Corollary 1 (Incremental Repair) .
For multiple-region repair, the patch for a new region A (cid:48) would not cause a previous patched region A to become incorrect.
Proof.
After applying the patch to linear region A , we have that the resulting network is correct on
A .
When applying a new patch to another linear region A (cid:48) , by Theorem 3, the new patch would not make a correct linear region A incorrect.
Theorem 1 (Soundness) .
The repaired DNN (cid:98) f returned by REASSURE is guaranteed to satisfy the speciﬁcation Φ .
Proof.
The proof has two parts:
1. to show that (cid:98) f satisﬁes the speciﬁcation Φ on A , and 2. to show that (cid:98) f satisﬁes the speciﬁcation Φ outside of A .
Part 1:
Lemma (2) shows (cid:98) f satisfy the speciﬁcation Φ for single-region repair on A .
For the multi-region case, consider a set of buggy linear regions ∪ 1 ≤ l ≤ I A l with the corresponding support neural network g A l and afﬁne patch function p
A l for each A l .
For the multi-region repair construction in Equation (8), we refer to σ ( p A j − p
A j − 1 +max k ≥
j
{ g A k } K j − K j ) as the j -th patch and (cid:98) f
j = f + (cid:80) j (cid:48)
≤ j σ ( p A j (cid:48) − p
A j (cid:48)− 1 + max k ≥ j (cid:48) { g A k } K j − K j ) as the network after the j -th patch.
For any x in patch area ∪ 1 ≤ l ≤ I A l , we can ﬁnd a j such that x ∈
A j but x / ∈
A k for all k > j .
After the ﬁrst j patches σ ( p A 1 ( x ) + max k ≥ 1 { g A k ( x, γ ) } K 1 − K 1 ) , σ ( p A 2 ( x ) − p
A 1 ( x ) + max k ≥ 2 { g A k ( x, γ ) } K 2 − K 2 ) , ... , σ ( p A j ( x ) − p
A j − 1 ( x ) + max k ≥ j { g A k ( x, γ ) } K j − K j ) , the
DNN’s output at x becomes (cid:98) f
j ( x ) = f
( x ) + p
A j ( x ) which meets our speciﬁcation Φ at x by the deﬁnition of p A j ( x ) .
Since x / ∈ A k for all k > j , then by Corollary 1, the rest of the patches would not change a correct area to an incorrect area.
Therefore, we have the ﬁnal patched neural network (cid:98) f meets speciﬁcation Φ on ∪ 1 ≤ l ≤ I A l .
Part 2:
To show that (cid:98) f satisﬁes Φ outside of A .
For any x outside the patch area ∪ 1 ≤ l ≤ I A l , we have x lies on a correct linear region (linear region that satisﬁes the speciﬁcation Φ ).
By Theorem 3, we have either (cid:98) f ( x ) =
f
( x ) or (cid:98) f ( x ) ∈
Φ out .
Therefore, (cid:98) f satisﬁes Φ outside of A .
Theorem 4 (Minimum Repair) .
For any ReLU DNN ˜ f , which is linear on a patch region A and satisﬁes the speciﬁcation Φ , there exists a positive number Γ , such that for all γ ≥ Γ ,
[Equation]
Proof.
We consider the general case where the linear patch function is obtained from Equation (3).
For any DNN ˜ f , which is linear on patch region A and satisﬁes the speciﬁcation Φ , we have
[Equation]
on patch area A by Equation (3).
Therefore, we only need to show:
[Equation]
Since parameter γ controls the slope of h A ( ., γ ) outside of patch area A , a large γ means that h A ( ., γ ) will drop to zero quickly outside of A .
Therefore, we can choose a large enough Γ such that h A ( ., γ ) drops to zero faster than the change of linear patch function c x + d .
Therefore, we have that for any γ ≥ Γ ,
[Equation]
Theorem 5 (Polynomial-Time Efﬁciency) .
REASSURE terminates in polynomial-time in the size of the neural network and the number of buggy linear regions when Φ out takes the form of { y | q l ≤
Py ≤ q u } , where P is a full row rank matrix and −∞ ≤ q l [ i ] ≤ q u [ i ] ≤ + ∞ ( q l [ i ] and q u [ i ] are the i -th elements of q l and q u respectively).
Proof.
We consider the afﬁne patch function solved via Equation (10).
Suppose
A = { x ∈ X | a i x ≤ b i , i ∈ I } .
For the LPs in Equation (11), | I | is the number of constraints and it is polynomial in the size of the neural network.
Thus, REASSURE runs in polynomial time in the size of the neural network.
In addition, since REASSURE computes the support network g A and afﬁne patch function p A for each A one by one (see Algorithm 1), the time complexity of REASSURE is linear in the number of buggy linear regions.
7. A DDITIONAL E XPERIMENT D ETAILS Details
on Feature-Space Repairs:
For an R -layer DNN f , we split f into two submodels f 1 and f 2 according to a hidden layer, say the j th hidden layer, where f 1 is the ﬁrst j layers function, f 2 is the last R − j layers function and f = f 2 ◦ f 1 .
We note the output space of f 1 the feature space.
And for any buggy input (cid:101) x , we note f 1 ( (cid:101) x ) the buggy feature.
Repairing in a feature space is to repair the behavior of f 2 on buggy features { f 1 ( (cid:101) x 1 ) , . . .
, f 1 ( (cid:101) x L ) } .
Note this will automatically repair the behavior of f on buggy points { (cid:101) x 1 , . . .
, (cid:101) x L } .
Repairing in a feature space has the beneﬁt of making the repair process more computation-friendly and reducing the parameter overhead of the additional networks, and has the potential to generalize the repair to undetected buggy inputs with similar features.
However, it loses the locality guarantee
[Table]
[Table]
Point-wise Repair on ImageNet (Feature-Space Repairs)
We use AlexNet Krizhevsky et al.
(2012) on ImageNet
dataset Russakovsky et al.
(2015) as the target DNN.
The size of image is (224, 224, 3) and the total number of classes for ImageNet is 1000.
We slightly modiﬁed AlexNet: we only consider 10 output classes that our buggy images may lie on and use a multilayer perceptron with three hidden layers (512, 256, 256 nodes respectively) to mimic the last two layers of AlexNet.
The goal of the repair is to ﬁx the behaviors of the target DNN on buggy inputs, which are found on ImageNet-A Hendrycks et al.
For REASSURE , we construct the patch network starting from the third from the last hidden layer (i.e. repair in a feature space).
The results are shown in Table 4.
REASSURE , PRDNN and Fine-Tuning repair all the buggy points while Retrain only repair 96% buggy points in 100 epochs.
REASSURE achieves almost zero modiﬁcation on validation images to the original DNN.
In addition, REASSURE preserves the performance of the original DNN.
Watermark Removal
We compare REASSURE with MDNN on the watermark removal experiment from their paper.
We were not able to run the code provided in the MDNN Github repository, but we were able to run on the target DNN models, watermark images, and MDNN-repaired models in the same repository.
The target DNN is from Goldberger et al.
(2020), which is watermarked by the method proposed in
Adi et al.
(2018) on a set of randomly chosen images x i with label f ( x i ) .
The goal is to change the DNN’s predictions on all watermarks x i to any other label y (cid:54) =
f ( x i ) while preserving the DNN’s performance on the MNIST test data.
For REASSURE , we set the prediction y =
f ( x i ) − 1 if f ( x i ) > 1 , and y = 10 otherwise.
The results are shown in Table 5.
Both REASSURE and MDNN remove all the watermarks.
However, MDNN introduces signiﬁcant distortion to the target DNN and as a result the test accuracy drops rapidly as the number of repair points increases.
In comparison, REASSURE removes all the watermarks with no harm to test accuracy.
[Table]
Area Repair: HCAS
Table 6 is the comparison with PRDNN using the last layer as the repair layer.
All other settings are the same as those in Section 5.2.
[Table]
The test accuracy of the original DNN is 97 .
#A: number of buggy linear regions to repair; random sampled input constraints
Speciﬁcation is repaired to incorrect in seconds.
For PRDNN, the ﬁrst running time is for enumerating all vertices of the polytopes the second is for solving the LP problem in PRDNN.
Size of Neural Networks:
Point-wise Repairs on MNIST:
The DNN model is a multilayer perceptron with ReLU activation functions.
It has an input layer with 784 nodes, 2 hidden layers with 256 nodes in each layer, and a ﬁnal output layer with 10 nodes.
Watermark Removal on MNIST:
The DNN model has an input layer with 784 nodes, a single hidden layer with 150 nodes, and a ﬁnal output layer with 10 nodes.
Area Repair on HCAS: The DNN model has an input layer with 3 nodes, 5 hidden layers withnodes in each hidden layer, and a ﬁnal output layer with 5 nodes.
DNN outputs one of ﬁve possible control advisories (’wrong left’, ’weak left’, ’Clear-of-Conﬂict’, ’weak right’ and ’wrong right’).
Point-wise Repairs on ImageNet: modiﬁed AlexNet Krizhevsky et al.
(2012) has 650k neurons, consists of ﬁve convolutional layers, some of which are followed by max-pooling layers, and ﬁve fully-connected layers.
Hyperparameters used in Repair:
We set γ = 0 .
5 for Point-wise Repair on MNIST, γ = 0 .
02 for Watermark Removal, γ = 1 for Area
Repair: HCAS and γ = 0 .
0005 for Point-wise Repair on ImageNet.
We set learning rate to 10 − 3 for Retrain in the point-wise repair experiment.
We set learning rate to 10 − 2 and momentum to 0 .
9 for Fine-Tuning in the point-wise repair experiment.
PRDNN requires specifying a layer for weight modiﬁcation.
We use the ﬁrst hidden layer as the repair layer, which has the best performance in our experiment settings, unless otherwise speciﬁed.
Speciﬁcations in HCAS:
Speciﬁcation 1.
If the intruder is near and approaching from the left, the network advises “strong right.”
Input constraints: Φ in = { ( x, y, ψ ) | 10 ≤ x ≤ 5000 , 10 ≤ y ≤ 5000 , − π ≤ ψ ≤ − 1 / 2 π } .
Output constraint: f ( x, y, ψ ) 4 ≥ f ( x, y, ψ ) i for i = 0 , 1 , 2 , 3 .
We calculate all the linear regions for N 1 , 4 in the area Φ in of Speciﬁcation 1 and totally 165 linear regions are found, including 87 buggy linear regions (DNN did not meet the speciﬁcation) andcorrect linear regions (DNN meet the speciﬁcation).
Speciﬁcation 2.
If the intruder is near and approaching from the right, the network advises “strong left.”
Input constraints: Φ in = { ( x, y, ψ ) | 10 ≤ x ≤ 5000 , − 5000 ≤ y ≤ − 10 , 1 / 2 π ≤ ψ ≤ π } .
Output constraint: f ( x, y, ψ ) 0 ≥ f ( x, y, ψ ) i for i = 1 , 2 , 3 , 4 .
Also we calculate all the linear regions in the area Φ in of Speciﬁcation 2.
And 79 correct linear regions are found.
We will test if a repair will make those correct linear regions incorrect.
Point-wise Repairs vs. Area Repairs:
REASSURE automatically performs area repair on the point-wise repair experiments.
This means our area repair method scales well to high-dimensional polytopes (the input dimension of MNIST is 784 ) whereas PRDNN does not scale beyond 2D linear regions/polytopes.
Parameter Overhead for REASSURE :
REASSURE introduces an additional network, patch network, and as a result adds new parameters to the original network.
The number of new parameters depends on |
I | , which is the number of linear constraints for the H-representation of A .
We can remove redundant constraints in this representation in polynomial time to make the additional network smaller.
For the area repair experiment on
HCAS, the average number of constraints for one linear region is 3 .
28 and the average number of new parameters that REASSURE introduces is 66 .
As a comparison, the number of parameters in the original network is around 3000 and PRDNN doubles the number of parameters (as a result of the Decoupled DNN construction) regardless of the number of point-wise or area repairs.
A PPLYING REASSURE T O G ENERAL CPWL N ETWORKS
Recall the result that an R m → R function is representable by a ReLU DNN if and only if it is a continuous piecewise linear (CPWL) function Arora et al.
We use convolutional neural networks as an example to show how REASSURE can be applied to more general CPWL networks.
Convolutional neural networks (CNNs) are neural networks with convolution layers and maxpooling layers.
For simplicity, we assume the CNNs also use ReLU activation functions (but in general other
CPWL activation functions will also work).
The convolutional layers can be viewed as special linear layers.
The maxpooling layers can be converted to linear operations with ReLU activation functions as follows.
[Equation]
where σ is the ReLU activation function.
Thus, REASSURE can be used to repair CNNs as well.