Original: U NIVERSAL A PPROXIMATION WITH C ERTIFIED N ETWORKS
Revised: U NIVERSAL A PPROXIMATION WITH C ERTIFIED N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training neural networks to be certiﬁably robust is critical to ensure their safety against adversarial attacks.
Revised: Training neural networks to be certiﬁably robust is critical to ensure their safety against adversarial attacks.

Original: However, while promising, state-of-the-art results with certiﬁed training are far from satisfactory.

Original: Currently, it is  very difﬁcult to train a neural network that is both accurate and certiﬁably robust.
Revised: However, it is currently very difﬁcult to train a neural network that is both accurate and certiﬁably robust.

Original: In this work we take a step towards addressing this challenge.
Revised: In this work we take a step towards addressing this challenge.

Original: We prove that for every continuous function f , there exists a network n such that: (i) n approximates f arbitrarily close, and (ii) simple interval bound propagation of a region B through n yields a result that is arbitrarily close to the optimal output of f on B .
Revised: We prove that for every continuous function f , there exists a network n such that: (i) n approximates f arbitrarily close and (ii) simple interval bound propagation of a region B through n yields a result that is arbitrarily close to the optimal output of f on B .

Original: Our result can be seen as a
Revised: Our result can be seen as a

Original: Universal Approximation Theorem for interval-certiﬁed ReLU networks.
Revised: Universal Approximation Theorem for interval-certiﬁed ReLU networks.

Original: To the best of our knowledge, this is the ﬁrst work to prove the existence of accurate, interval-certiﬁed networks.
Revised: To the best of our knowledge, this is the ﬁrst work to prove the existence of accurate, interval-certiﬁed networks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Much recent work has shown that neural networks can be fooled into misclassifying adversarial examples (Szegedy et al., 2014), inputs which are imperceptibly different from those that the neural network classiﬁes correctly.
Revised: Much recent work has shown that neural networks can be fooled into misclassifying adversarial examples (Szegedy et al., 2014), inputs which are imperceptibly different from those that the neural network classiﬁes correctly.

Original: Initial work on defending against adversarial examples revolved around training networks to be empirically robust, usually by including adversarial examples found with various attacks into the training dataset (Gu and Rigazio, 2015; Papernot et al., 2016; Zheng et al., 2016; Athalye et al., 2018; Eykholt et al., 2018; Moosavi-Dezfooli et al., 2017; Xiao et al., 2018).
Revised: Initial work on defending against adversarial examples revolved around training networks to be empirically robust, usually by including adversarial examples found with various attacks into the training dataset (Gu and Rigazio, 2015; Papernot et al., 2016; Zheng et al., 2016; Athalye et al., 2018; Eykholt et al., 2018; Moosavi-Dezfooli et al., 2017; Xiao et al., 2018).

Original: However, while empirical robustness can be practically useful, it does not provide safety guarantees.
Revised: However, while empirical robustness can be practically useful, it does not provide safety guarantees.

Original: As a result, much recent research has focused on verifying that a network is certiﬁably robust, typically by employing methods based on mixed integer linear programming (Tjeng et al., 2019),
Revised: As a result, much recent research has focused on verifying that a network is certiﬁably robust, typically by employing methods based on mixed integer linear programming (Tjeng et al., 2019),

Original: SMT solvers (Katz et al., 2017), semideﬁnite programming (Raghunathan et al., 2018a), duality (Wong and Kolter, 2018; Krishnamurthy et al., 2018), and linear relaxations (Gehr et al., 2018; 
Revised: SMT solvers (Katz et al., 2017), semideﬁnite programming (Raghunathan et al., 2018a), duality (Wong and Kolter, 2018; Dvijotham et al., 2018b), and linear relaxations (Gehr et al., 2018; Weng et al., 2018;

Original: Weng et al., 2018; Wang et al., 2018b; Zhang et al., 2018; Singh et al., 2018;
Revised: Wang et al., 2018b; Zhang et al., 2018; Singh et al., 2018;

Original: Salman et al., 2019).
Revised: Salman et al., 2019).

Original: Because the certiﬁcation rates were far from satisfactory, speciﬁc training methods were recently developed which produce networks that are certiﬁably robust: Mirman et al.
Revised: Because the certiﬁcation rates were far from satisfactory, speciﬁc training methods were recently developed which produce networks that are certiﬁably robust: Mirman et al.

Original: ; Raghunathan et al.
Revised: ; Raghunathan et al.

Original: (2018b)
Revised: (2018b)

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2018a)
Revised: (2018a)

Original: ; Wong and Kolter (2018); Wong et al.
Revised: ; Wong and Kolter (2018); Wong et al.

Original: ; Gowal et al.
Revised: ; Gowal et al.

Original: (2018) train the network with standard optimization applied to an over-approximation of the network behavior on a given input region (the region is created around the concrete input point).
Revised: (2018) train the network with standard optimization applied to an over-approximation of the network behavior on a given input region (the region is created around the concrete input point).

Original: These techniques aim to discover speciﬁc weights which facilitate veriﬁcation.
Revised: These techniques aim to discover speciﬁc weights which facilitate veriﬁcation.

Original: There is a tradeoff between the degree of the over-approximation used and the speed of training and certiﬁcation.
Revised: There is a tradeoff between the degree of the over-approximation used and the speed of training and certiﬁcation.

Original: Recently, (Cohen et al., 2019b) proposed a statistical approach to certiﬁcation, which unlike the non-probabilistic methods discussed above, creates a probabilistic classiﬁer that comes with probabilistic guarantees.
Revised: Recently, (Cohen et al., 2019b) proposed a statistical approach to certiﬁcation, which unlike the non-probabilistic methods discussed above, creates a probabilistic classiﬁer that comes with probabilistic guarantees.

Original: So far, some of the best non-probabilistic results achieved on the popular MNIST (Lecun et al., 1998) and CIFAR10 (Krizhevsky, 2009) datasets have been obtained with the simple Interval relaxation (Gowal et al., 2018; Mirman et al., 2019), which scales well at both training and veriﬁcation time.
Revised: So far, some of the best non-probabilistic results achieved on the popular MNIST (Lecun et al., 1998) and CIFAR10 (Krizhevsky, 2009) datasets have been obtained with the simple Interval relaxation (Gowal et al., 2018; Mirman et al., 2019), which scales well at both training and veriﬁcation time.

Original: Despite this progress, there are still substantial gaps between known standard accuracy, experimental robustness, and certiﬁed robustness.
Revised: Despite this progress, there are still substantial gaps between known standard accuracy, experimental robustness, and certiﬁed robustness.

Original: For example, for CIFAR10, the best reported certiﬁed robustness is 32.04% with an accuracy of 49.49% when using a fairly modest l ∞ region with radius 8/255 (Gowal et al., 2018).
Revised: For example, for CIFAR10, the best reported certiﬁed robustness is 32.04% with an accuracy of 49.49% when using a fairly modest l ∞ region with radius 8/255 (Gowal et al., 2018).

Original: The state-of-the-art non-robust accuracy for this dataset is > 95% with experimental robustness > 50%.
Revised: The state-of-the-art non-robust accuracy for this dataset is > 95% with experimental robustness > 50%.

Original: Given the size of this gap, a key question then is: can certiﬁed training ever succeed or is there a fundamental limit ?
Revised: Given the size of this gap, a key question then is: can certiﬁed training ever succeed or is there a fundamental limit ?

Original: x    y  −−   1 [0 , 1] [0 , 12 ] [0 , 12 ] [0 , 34 ] [0 , 34 ] [0 , 32 ] (a) Not certiﬁable network n 1 .
Revised: x    y  −−   1 [0 , 1] [0 , 12 ] [0 , 12 ] [0 , 34 ] [0 , 34 ] [0 , 32 ] (a) Not certiﬁable network n 1 .

Original: 1 x 0 .0 .1 .y
Revised: 1 x 0 .0 .1 .y

Original: f
Revised: f

Original: ( x ) (b) The function f .
Revised: ( x ) (b) The function f .

Original: x    y  −−−− 1 [0 , 1] [0 , 12 ] [0 , 12 ] [0 , 12 ] [0 , 12 ] [0 , 1] (c) Certiﬁable network n 2 .
Revised: x    y  −−−− 1 [0 , 1] [0 , 12 ] [0 , 12 ] [0 , 12 ] [0 , 12 ] [0 , 1] (c) Certiﬁable network n 2 .

Original: In this paper we take a step in answering this question by proving a result parallel to the Universal Approximation Theorem (Cybenko, 1989;
Revised: In this paper we take a step in answering this question by proving a result parallel to the Universal Approximation Theorem (Cybenko, 1989;

Original: Hornik et al., 1989).
Revised: Hornik et al., 1989).

Original: We prove that for any continuous function f deﬁned on a compact domain Γ ⊆
Revised: We prove that for any continuous function f deﬁned on a compact domain Γ ⊆

Original: R m and for any desired level of accuracy δ , there exists a ReLU neural network n which can certiﬁably approximate f up to δ using interval bound propagation.
Revised: R m and for any desired level of accuracy δ , there exists a ReLU neural network n which can certiﬁably approximate f up to δ using interval bound propagation.

Original: As an interval is a fairly imprecise relaxation, our result directly applies to more precise convex relaxations (e.g.,
Revised: As an interval is a fairly imprecise relaxation, our result directly applies to more precise convex relaxations (e.g.,

Original: Zhang et al.
Revised: Zhang et al.

Original: ; Singh et al.
Revised: ; Singh et al.

Original: Theorem 1.1 (Universal Interval-Certiﬁed Approximation, Figure 1) .
Revised: Theorem 1.1 (Universal Interval-Certiﬁed Approximation, Figure 1) .

Original: Let Γ ⊂
Revised: Let Γ ⊂

Original: R m be a compact set and let f : Γ → R be a continuous function.
Revised: R m be a compact set and let f : Γ → R be a continuous function.

Original: For all δ > 0 , there exists a ReLU network n such that for all boxes [ a, b ] in Γ deﬁned by points a, b ∈ Γ where a k ≤ b k for all k , the propagation of the box [ a, b ] using interval analysis through the network n , denoted n (cid:93) ([ a, b ]) , approximates the set [ l, u ] =
Revised: For all δ > 0 , there exists a ReLU network n such that for all boxes [ a, b ] in Γ deﬁned by points a, b ∈ Γ where a k ≤ b k for all k , the propagation of the box [ a, b ] using interval analysis through the network n , denoted n (cid:93) ([ a, b ]) , approximates the set [ l, u ] =

Original: [min f ([ a, b ]) , max f ([ a, b ])] ⊆
Revised: [min f ([ a, b ]) , max f ([ a, b ])] ⊆

Original: R up to δ ,
Revised: R up to δ ,

Original: [Equation]
Revised: [Equation]

Original: We recover the classical universal approximation theorem ( | f ( x ) − n ( x ) | ≤ δ for all x ∈ Γ ) by considering boxes [ a, b ] describing points ( x = a = b ).
Revised: We recover the classical universal approximation theorem ( | f ( x ) − n ( x ) | ≤ δ for all x ∈ Γ ) by considering boxes [ a, b ] describing points ( x = a = b ).

Original: Note that here the lower bound is not [ l, u ] as the network n is an approximation of f .
Revised: Note that here the lower bound is not [ l, u ] as the network n is an approximation of f .

Original: Because interval analysis propagates boxes, the theorem naturally handles l ∞ norm bound perturbations to the input.
Revised: Because interval analysis propagates boxes, the theorem naturally handles l ∞ norm bound perturbations to the input.

Original: Other l p norms can be handled by covering the l p ball with boxes.
Revised: Other l p norms can be handled by covering the l p ball with boxes.

Original: The theorem can be extended easily to functions f : Γ → R k by applying the theorem component wise.
Revised: The theorem can be extended easily to functions f : Γ → R k by applying the theorem component wise.

Original: Practical meaning of theorem The practical meaning of this theorem is as follows: if we train a neural network n (cid:48) on a given training data set (e.g., CIFAR10) and we are satisﬁed with the properties of n (cid:48) (e.g., high accuracy), then because n (cid:48) is a continuous function, the theorem tells us that there exists a network n which is as accurate as n (cid:48) and as certiﬁable with interval analysis as n (cid:48) is with a complete veriﬁer.
Revised: Practical meaning of theorem The practical meaning of this theorem is as follows: if we train a neural network n (cid:48) on a given training data set (e.g., CIFAR10) and we are satisﬁed with the properties of n (cid:48) (e.g., high accuracy), then because n (cid:48) is a continuous function, the theorem tells us that there exists a network n which is as accurate as n (cid:48) and as certiﬁable with interval analysis as n (cid:48) is with a complete veriﬁer.

Original: This means that if we fail to ﬁnd such an n , then either n did not possess the required capacity or the optimizer was unsuccessful.
Revised: This means that if we fail to ﬁnd such an n , then either n did not possess the required capacity or the optimizer was unsuccessful.

Original: Focus on the existence of a network We note that we do not provide a method for training a certiﬁed ReLU network – even though our method is constructive , we aim to answer an existential question and thus we focus on proving that a given network exists.
Revised: Focus on the existence of a network We note that we do not provide a method for training a certiﬁed ReLU network – even though our method is constructive , we aim to answer an existential question and thus we focus on proving that a given network exists.

Original: Interesting future work items would be to study the requirements on the size of this network and the inherent hardness of ﬁnding it with standard optimization methods.
Revised: Interesting future work items would be to study the requirements on the size of this network and the inherent hardness of ﬁnding it with standard optimization methods.

Original: Universal approximation is insufﬁcient We now discuss why classical universal approximation is insufﬁcient for establishing our result.
Revised: Universal approximation is insufﬁcient We now discuss why classical universal approximation is insufﬁcient for establishing our result.

Original: While classical universal approximation theorems state that neural networks can approximate a large class of functions f , unlike our result, they do not state that robustness of the approximation n of f is actually certiﬁed with a scalable proof method (e.g., interval bound propagation).
Revised: While classical universal approximation theorems state that neural networks can approximate a large class of functions f , unlike our result, they do not state that robustness of the approximation n of f is actually certiﬁed with a scalable proof method (e.g., interval bound propagation).

Original: If one uses a non scalable complete veriﬁer instead, then the standard
Revised: If one uses a non scalable complete veriﬁer instead, then the standard

Original: Universal approximation theorem is sufﬁcient.
Revised: Universal approximation theorem is sufﬁcient.

Original: To demonstrate this point, consider the function f : R → R (Figure 2b) mapping all x ≤ 0 to 1 , all x ≥ 1 to 0 and all 0 < x < 1 to 1 − x and two ReLU networks n 1 (Figure 2a) and n 2 (Figure 2c) perfectly approximating f , that is n 1 ( x )
Revised: To demonstrate this point, consider the function f : R → R (Figure 2b) mapping all x ≤ 0 to 1 , all x ≥ 1 to 0 and all 0 < x < 1 to 1 − x and two ReLU networks n 1 (Figure 2a) and n 2 (Figure 2c) perfectly approximating f , that is n 1 ( x )

Original: = f
Revised: = f

Original: ( x ) = n 2 ( x ) for all x .
Revised: ( x ) = n 2 ( x ) for all x .

Original: For δ = 14 , the interval certiﬁcation that n 1 maps all x ∈ [0 , 1] to [0 , 1] fails because [ 14 , 34 ] ⊆ n (cid:93) 1 ([0 , 1]) =
Revised: For δ = 14 , the interval certiﬁcation that n 1 maps all x ∈ [0 , 1] to [0 , 1] fails because [ 14 , 34 ] ⊆ n (cid:93) 1 ([0 , 1]) =

Original: [0 , 32 ] (cid:54)⊆ [ − 14 , 54 ] .
Revised: [0 , 32 ] (cid:54)⊆ [ − 14 , 54 ] .

Original: However, interval certiﬁcation succeeds for n 2 , because n (cid:93) 2 ([0 , 1]) =
Revised: However, interval certiﬁcation succeeds for n 2 , because n (cid:93) 2 ([0 , 1]) =

Original: To the best of our knowledge, this is the ﬁrst work to prove the existence of accurate, interval-certiﬁed networks.
Revised: To the best of our knowledge, this is the ﬁrst work to prove the existence of accurate, interval-certiﬁed networks.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: After adversarial examples were discovered by Szegedy et al.
Revised: After adversarial examples were discovered by Szegedy et al.

Original: (2014), many attacks and defenses were introduced (for a survey, see Akhtar and Mian (2018)).
Revised: (2014), many attacks and defenses were introduced (for a survey, see Akhtar and Mian (2018)).

Original: Initial work on verifying neural network robustness used exact methods (Katz et al., 2017; Tjeng et al., 2019) on small networks, while later research introduced methods based on over-approximation (Gehr et al., 2018; Raghunathan et al., 2018a; Singh et al., 2018;
Revised: Initial work on verifying neural network robustness used exact methods (Katz et al., 2017; Tjeng et al., 2019) on small networks, while later research introduced methods based on over-approximation (Gehr et al., 2018; Raghunathan et al., 2018a; Singh et al., 2018;

Original: Salman et al., 2019) aiming to scale to larger networks.
Revised: Salman et al., 2019) aiming to scale to larger networks.

Original: A fundamentally different approach is randomized smoothing (Li et al., 2018;
Revised: A fundamentally different approach is randomized smoothing (Li et al., 2019;

Original: Lécuyer et al., 2019; Cohen et al., 2019b), in which probabilistic classiﬁcation and certiﬁcation with high conﬁdence is performed.
Revised: Lécuyer et al., 2019; Cohen et al., 2019b), in which probabilistic classiﬁcation and certiﬁcation with high conﬁdence is performed.

Original: As neural networks that are experimentally robust need not be certiﬁably robust, there has been signiﬁcant recent research on training certiﬁably robust neural networks (Raghunathan et al., 2018b;
Revised: As neural networks that are experimentally robust need not be certiﬁably robust, there has been signiﬁcant recent research on training certiﬁably robust neural networks (Raghunathan et al., 2018b;

Original: Mirman et al., 2018; 2019; Wong and Kolter, 2018; Wong et al., 2018;
Revised: Mirman et al., 2018; 2019; Wong and Kolter, 2018; Wong et al., 2018;

Original: Wang et al., 2018a; Gowal et al., 2018; Dvijotham et al., 2018; Xiao et al., 2019; Cohen et al., 2019b).
Revised: Wang et al., 2018a; Gowal et al., 2018; Dvijotham et al., 2018a; Xiao et al., 2019; Cohen et al., 2019b).

Original: As these methods appear to have reached a performance wall, several works have started investigating the fundamental barriers in the datasets and methods that preclude the learning of a robust network (let alone a certiﬁably robust one)
Revised: As these methods appear to have reached a performance wall, several works have started investigating the fundamental barriers in the datasets and methods that preclude the learning of a robust network (let alone a certiﬁably robust one)

Original: (Khoury and Hadﬁeld-Menell, 2018; Schmidt et al., 2018; Tsipras et al., 2019).
Revised: (Khoury and Hadﬁeld-Menell, 2018; Schmidt et al., 2018; Tsipras et al., 2019).

Original: In our work, we focus on the question of whether neural networks are capable of approximating functions whose robustness can be established with the efﬁcient interval relaxation.
Revised: In our work, we focus on the question of whether neural networks are capable of approximating functions whose robustness can be established with the efﬁcient interval relaxation.

Original: Feasibility Results with Neural Networks Early versions of the Universal Approximation Theorem were stated by Cybenko (1989) and Hornik et al.
Revised: Feasibility Results with Neural Networks Early versions of the Universal Approximation Theorem were stated by Cybenko (1989) and Hornik et al.

Original: Cybenko (1989) showed that networks using sigmoidal activations could approximate continuous functions in the unit hypercube, while
Revised: Cybenko (1989) showed that networks using sigmoidal activations could approximate continuous functions in the unit hypercube, while

Original: Hornik et al.
Revised: Hornik et al.

Original: (1989) showed that even networks with only one hidden layer are capable of approximating Borel measurable functions.
Revised: (1989) showed that even networks with only one hidden layer are capable of approximating Borel measurable functions.

Original: More recent work has investigated the capabilities of ReLU networks.
Revised: More recent work has investigated the capabilities of ReLU networks.

Original: Here, Arora et al.
Revised: Here, Arora et al.

Original: (2018), based on Tarela and Martínez (1999), proved that every continuous piecewise linear function in R m can be represented by a ReLU network.
Revised: (2018), based on Tarela and Martínez (1999), proved that every continuous piecewise linear function in R m can be represented by a ReLU network.

Original: Later, He et al.
Revised: Later, He et al.

Original: (2018) reduced the number of neurons needed using ideas from ﬁnite elements methods.
Revised: (2018) reduced the number of neurons needed using ideas from ﬁnite elements methods.

Original: Relevant to our work, Arora et al.
Revised: Relevant to our work, Arora et al.

Original: (2018) introduced a
Revised: (2018) introduced a

Original: ReLU network representations of the min function.
Revised: ReLU network representations of the min function.

Original: Further, we use a construction method that is similar to the construction for nodal basis functions given in He et al.
Revised: Further, we use a construction method that is similar to the construction for nodal basis functions given in He et al.

Original: Universal approximation for Lipschitz constrained networks have been considered by Anil et al.
Revised: Universal approximation for Lipschitz constrained networks have been considered by Anil et al.

Original: (2018) and later by Cohen et al. (2019a).
Revised: (2019) and later by Cohen et al. (2019a).

Original: A bound on the Lipschitz constant of a network immediately yields a certiﬁed region depending on the classiﬁcation margin.
Revised: A bound on the Lipschitz constant of a network immediately yields a certiﬁed region depending on the classiﬁcation margin.

Original: Anil et al.
Revised: Anil et al.

Original: (2018) proved that the set of Lipschitz networks with the GroupSort activation is dense in the space of Lipschitz continuous functions with Lipschitz constant 1, while Cohen et al.
Revised: (2019) proved that the set of Lipschitz networks with the GroupSort activation is dense in the space of Lipschitz continuous functions with Lipschitz constant 1, while Cohen et al.

Original: (2019a) provide an explicit construction to obtain the network.
Revised: (2019a) provide an explicit construction to obtain the network.

Original: We note that both of these works focus on Lipschitz continuous functions, a more restricted class than continuous functions, which we consider in our work.
Revised: We note that both of these works focus on Lipschitz continuous functions, a more restricted class than continuous functions, which we consider in our work.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section we provide the concepts necessary to describe our main result.
Revised: In this section we provide the concepts necessary to describe our main result.

Original: Adversarial Examples and Robustness Veriﬁcation Let n : R m → R k be a neural network, which classiﬁes an input x to a label t if n ( x ) t > n ( x ) j for all j (cid:54) =
Revised: Adversarial Examples and Robustness Veriﬁcation Let n : R m → R k be a neural network, which classiﬁes an input x to a label t if n ( x ) t > n ( x ) j for all j (cid:54) =

Original: t .
Revised: t .

Original: For a correctly classiﬁed input x , an adversarial example is an input y such that x is imperceptible from y to a human, but is classiﬁed to a different label by n .
Revised: For a correctly classiﬁed input x , an adversarial example is an input y such that x is imperceptible from y to a human, but is classiﬁed to a different label by n .

Original: Frequently, two images are assumed to be “imperceptible” if there l p distance is at most (cid:15) .
Revised: Frequently, two images are assumed to be “imperceptible” if there l p distance is at most (cid:15) .

Original: The l p ball around an image is said to be the adversarial ball, and a network is said to be (cid:15)
Revised: The l p ball around an image is said to be the adversarial ball, and a network is said to be (cid:15)

Original: -robust around x if
Revised: -robust around x if

Original: -  2 x - -6/ -2/ 2/ 6/  y (a) f 4 / 4 / 4 / 5 y 4 /-
Revised: -  2 x - -6/ -2/ 2/ 6/  y (a) f 4 / 4 / 4 / 5 y 4 /-

Original:  2 x 4 /( [Figure]
Revised:  2 x 4 /(b)

Original: Slicing of f , f 0 , . .
Revised: Slicing of f , f 0 , . .

Original: , f     1 y2 x  ( [Figure] approximating f k .
Revised: , f     1 y2 x  (c) Networks n k approximating f k .

Original: n k (Figure 3c) approximate the N -slicing of f (Figure 3b), as a sum of local bumps (Figure 6).
Revised: n k (Figure 3c) approximate the N -slicing of f (Figure 3b), as a sum of local bumps (Figure 6).

Original: every point in the adversarial ball around x classiﬁes the same.
Revised: every point in the adversarial ball around x classiﬁes the same.

Original: In this paper, we limit our discussion to l ∞ adversarial balls which can be used to cover to all l p balls.
Revised: In this paper, we limit our discussion to l ∞ adversarial balls which can be used to cover to all l p balls.

Original: The goal of robustness veriﬁcation is to show that for a neural network n , input point x and label t , every possible input in an l ∞ ball of size (cid:15) around x (written B ∞ (cid:15) ( x ) ) is also classiﬁed to t .
Revised: The goal of robustness veriﬁcation is to show that for a neural network n , input point x and label t , every possible input in an l ∞ ball of size (cid:15) around x (written B ∞ (cid:15) ( x ) ) is also classiﬁed to t .

Original: Verifying neural networks with Interval Analysis
Revised: Verifying neural networks with Interval Analysis

Original: The veriﬁcation technique we investigate in this work is interval analysis.
Revised: The veriﬁcation technique we investigate in this work is interval analysis.

Original: We denote by B the set of boxes B =
Revised: We denote by B the set of boxes B =

Original: [ a, b ] ⊂ R m for all m , where a i ≤ b i for all i .
Revised: [ a, b ] ⊂ R m for all m , where a i ≤ b i for all i .

Original: Furthermore for Γ ⊆
Revised: Furthermore for Γ ⊆

Original: R m we deﬁne B (Γ) :
Revised: R m we deﬁne B (Γ) :

Original: = B ∩ Γ describing all the boxes in Γ .
Revised: = B ∩ Γ describing all the boxes in Γ .

Original: The standard interval-transformations for the basic operations we are considering, namely + , − , · and the ReLU function R (Gehr et al.
Revised: The standard interval-transformations for the basic operations we are considering, namely + , − , · and the ReLU function R (Gehr et al.

Original: (2018), Gowal et al.
Revised: (2018), Gowal et al.

Original: (2018)) are [ a, b ] + (cid:93) [ c, d ] =
Revised: (2018)) are [ a, b ] + (cid:93) [ c, d ] =

Original: [ a + c, b + d ]
Revised: [ a + c, b + d ]

Original: R (cid:93) ([ a, b ]) =
Revised: R (cid:93) ([ a, b ]) =

Original: [ R ( a ) , R ( b )] −
Revised: [ R ( a ) , R ( b )] −

Original: [Equation]
Revised: [Equation]

Original: where [ a, b ] , [ c, d ] ∈ B ( R ) , and λ ∈ R ≥ 0 .
Revised: where [ a, b ] , [ c, d ] ∈ B ( R ) , and λ ∈ R ≥ 0 .

Original: Furthermore, we used (cid:93) to distinguish the function f from its interval-transformation f (cid:93) .
Revised: Furthermore, we used (cid:93) to distinguish the function f from its interval-transformation f (cid:93) .

Original: To illustrate the difference between f and f (cid:93) , consider f ( x ) :
Revised: To illustrate the difference between f and f (cid:93) , consider f ( x ) :

Original: = x − x evaluated on x = [0 , 1] .
Revised: = x − x evaluated on x = [0 , 1] .

Original: We have f ([0 , 1]) = 0 , but f (cid:93) ([0 , 1]) =
Revised: We have f ([0 , 1]) = 0 , but f (cid:93) ([0 , 1]) =

Original: [0 , 1] − # [0 , 1] =
Revised: [0 , 1] − # [0 , 1] =

Original: [0 , 1]+ # [ − 1 , 0] =
Revised: [0 , 1]+ # [ − 1 , 0] =

Original: [ − 1 , 1] illustrating the loss in precision that interval analysis suffers from.
Revised: [ − 1 , 1] illustrating the loss in precision that interval analysis suffers from.

Original: Interval analysis provides a sound over-approximation in the sense that for all function f , the values that f can obtain on [ a, b ] , namely f ([ a, b ]) :
Revised: Interval analysis provides a sound over-approximation in the sense that for all function f , the values that f can obtain on [ a, b ] , namely f ([ a, b ]) :

Original: = { f ( x ) | x ∈ [ a, b ] } are a subset of f (cid:93) ([ a, b ]) .
Revised: = { f ( x ) | x ∈ [ a, b ] } are a subset of f (cid:93) ([ a, b ]) .

Original: If f is a composition of functions, f = f 1 ◦ · · · ◦ f k , then f (cid:93) 1 ◦ · · · ◦ f
Revised: If f is a composition of functions, f = f 1 ◦ · · · ◦ f k , then f (cid:93) 1 ◦ · · · ◦ f

Original: (cid:93)k is a sound interval-transformer for f .
Revised: (cid:93)k is a sound interval-transformer for f .

Original: Furthermore all combinations f of + , − , · and R are monotone, that is for [ a, b ] , [ c, d ] ⊆ B ( R m )
Revised: Furthermore all combinations f of + , − , · and R are monotone, that is for [ a, b ] , [ c, d ] ⊆ B ( R m )

Original: such that [ a, b ] ⊆
Revised: such that [ a, b ] ⊆

Original: [ c, d ] then f # ([ a, b ]) ⊆
Revised: [ c, d ] then f # ([ a, b ]) ⊆

Original: f # ([ c, d ]) (Appendix A).
Revised: f # ([ c, d ]) (Appendix A).

Original: For boxes [ x, x ] representing points f (cid:93) coincides with f , f (cid:93) (
Revised: For boxes [ x, x ] representing points f (cid:93) coincides with f , f (cid:93) (

Original: [ x, x ]) =
Revised: [ x, x ]) =

Original: f ( x ) .
Revised: f ( x ) .

Original: This will later be needed.
Revised: This will later be needed.

Original:  P ROVING U NIVERSAL I NTERVAL -P ROVABLE A PPROXIMATION
Revised:  P ROVING U NIVERSAL I NTERVAL -P ROVABLE A PPROXIMATION

Original: In this section, we provide an explanation of the proof of our main result, Theorem 4.6, and illustrate the main points of the proof.
Revised: In this section, we provide an explanation of the proof of our main result, Theorem 4.6, and illustrate the main points of the proof.

Original: The full proof can be found in Appendix A.

Original: The ﬁrst step in the construction is to deconstruct the function f into slices { f k : Γ → [0 , δ 2 ] } 0 ≤ k<N such that that f ( x ) =
Revised: The ﬁrst step in the construction is to deconstruct the function f into slices { f k : Γ → [0 , δ 2 ] } 0 ≤ k<N such that that f ( x ) =

Original: ξ 0 + (cid:80) N − 1 k =

Revised: ξ 0 + (cid:80) N − 1 k =

Original: f k ( x ) for all x , where ξ 0 is the minimum of f (Γ) .
Revised: f k ( x ) for all x , where ξ 0 is the minimum of f (Γ) .

Original: We approximate each slice f k by a ReLU network δ 2 · n k .
Revised: We approximate each slice f k by a ReLU network δ 2 · n k .

Original: The network n approximating f up to δ will be n ( x ) :
Revised: The network n approximating f up to δ will be n ( x ) :

Original: =
Revised: =

Original: ξ 

Revised: ξ

Original: + δ 2 (cid:80) k n k ( x ) .
Revised: + δ 2 (cid:80) k n k ( x ) .

Original: The construction relies on 2 key insights, (i) the output of δ 2 · n (cid:93)k can be conﬁned to the interval [0 , δ 2 ] , thus the loss of analysis precision is at most the height of the slice, and (ii) we can construct the networks n k using local bump functions, such that only 4 slices can contribute to the loss of analysis precision, two for the lower interval bound, two for the upper one.
Revised: The construction relies on 2 key insights, (i) the output of δ 2 · n (cid:93)k can be conﬁned to the interval [0 , δ 2 ] , thus the loss of analysis precision is at most the height of the slice, and (ii) we can construct the networks n k using local bump functions, such that only 4 slices can contribute to the loss of analysis precision, two for the lower interval bound, two for the upper one.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The slicing { f k } 0 ≤ k< 5 of the function f : [ − 2 , 2] → R (Figure 3a), mapping x to f ( x ) = − x 3 + 3 x is depicted in Figure 3b.
Revised: The slicing { f k } 0 ≤ k< 5 of the function f : [ − 2 , 2] → R (Figure 3a), mapping x to f ( x ) = − x 3 + 3 x is depicted in Figure 3b.

Original: The networks n k are depicted in Figure 3c.
Revised: The networks n k are depicted in Figure 3c.

Original: In this example, evaluating the interval-transformer of n , namely n (cid:93) on the box B = [ − 1 , 1] results into n (cid:93) ([ − 1 , 1]) =
Revised: In this example, evaluating the interval-transformer of n , namely n (cid:93) on the box B = [ − 1 , 1] results into n (cid:93) ([ − 1 , 1]) =

Original: [ − 2 , 6 / 5] lies is within the δ = 85 bound of f ([ − 1 , 1]) =
Revised: [ − 2 , 6 / 5] lies is within the δ = 85 bound of f ([ − 1 , 1]) =

Original: [ − 2 , 2] .
Revised: [ − 2 , 2] .

Original: Deﬁnition 4.1 ( N -slicing (Figure 3b)) .
Revised: Deﬁnition 4.1 ( N -slicing (Figure 3b)) .

Original: Let Γ ⊂ R m be a closed m -dimensional box and let f : Γ →
Revised: Let Γ ⊂ R m be a closed m -dimensional box and let f : Γ →

Original: R be continuous.
Revised: R be continuous.

Original: The N -slicing of f is a set of functions { f k } 0 ≤ k<N deﬁned by
Revised: The N -slicing of f is a set of functions { f k } 0 ≤ k<N deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where ξ k := ξ 0 + kN ( ξ N − ξ 0 ) , k ∈ { 1 , . . .
Revised: where ξ k := ξ 0 + kN ( ξ N − ξ 0 ) , k ∈ { 1 , . . .

Original: , N − 1 } , ξ 0 :=
Revised: , N − 1 } , ξ 0 :=

Original: min f (Γ) and ξ N := max f (Γ) .
Revised: min f (Γ) and ξ N := max f (Γ) .

Original: To construct a ReLU network satisfying the desired approximation property (Equation (1)) if evaluated on boxes in B (Γ) , we need the ReLU network nmin capturing the behavior of min as a building block (similar to He et al.
Revised: To construct a ReLU network satisfying the desired approximation property (Equation (1)) if evaluated on boxes in B (Γ) , we need the ReLU network nmin capturing the behavior of min as a building block (similar to He et al.

Original: It is given by
Revised: It is given by

Original: [Equation]
Revised: [Equation]

Original: With the ReLU network nmin, we can construct recursively a ReLU network nmin N mapping
Revised: With the ReLU network nmin, we can construct recursively a ReLU network nmin N mapping

Original: N arguments to the smallest one (Deﬁnition A.8).
Revised: N arguments to the smallest one (Deﬁnition A.8).

Original: Even though the interval-transformation loses precision, we can establish bounds on the precision loss of nmin (cid:93)N sufﬁcient for our use case (Appendix A).
Revised: Even though the interval-transformation loses precision, we can establish bounds on the precision loss of nmin (cid:93)N sufﬁcient for our use case (Appendix A).

Original: Now, we use the clipping function R [ ∗ , 1] := 1 − R (1 − x ) clipping every value exceeding 1 back to 1 (Figure 5) to construct the local bumps φ c w.r.t.
Revised: Now, we use the clipping function R [ ∗ , 1] := 1 − R (1 − x ) clipping every value exceeding 1 back to 1 (Figure 5) to construct the local bumps φ c w.r.t.

Original: a grid G .
Revised: a grid G .

Original: G speciﬁes the set of all possible local bumps we can use to construct the networks n k .
Revised: G speciﬁes the set of all possible local bumps we can use to construct the networks n k .

Original: Increasing the ﬁnesse of G will increases the approximation precision.
Revised: Increasing the ﬁnesse of G will increases the approximation precision.

Original: Deﬁnition 4.2 (local bump, Figure 6) .
Revised: Deﬁnition 4.2 (local bump, Figure 6) .

Original: Let M ∈ N , G :=
Revised: Let M ∈ N , G :=

Original: { ( i 1 M ) , . .
Revised: { ( i 1 M ) , . .

Original: , i m M | i ∈
Revised: , i m M | i ∈

Original: Z m } be a grid, (cid:96) =
Revised: Z m } be a grid, (cid:96) =

Original: 2 (cid:100) log 2 2 m (cid:101) +1 and let c
Revised: 2 (cid:100) log 2 2 m (cid:101) +1 and let c

Original: = { i l 1 M , i u 1 M } × · · · × { i lm M , i um M } ⊆
Revised: = { i l 1 M , i u 1 M } × · · · × { i lm M , i um M } ⊆

Original: G be a set of grid points describing the corner points of a hyperrectangle in G .
Revised: G be a set of grid points describing the corner points of a hyperrectangle in G .

Original: We deﬁne a ReLU neural network φ c : R m → [0 , 1] ⊂ R w.r.t.
Revised: We deﬁne a ReLU neural network φ c : R m → [0 , 1] ⊂ R w.r.t.

Original: G by
Revised: G by

Original: [Equation]
Revised: [Equation]

Original: We will describe later how M and c get picked.
Revised: We will describe later how M and c get picked.

Original: A graphical illustration of a local bump for in two dimensions and c = { i l 1 M , i u 1 M } × { i l 2 M , i u 2 M } =
Revised: A graphical illustration of a local bump for in two dimensions and c = { i l 1 M , i u 1 M } × { i l 2 M , i u 2 M } =

Original: { c ll , c lu , c ul , c uu } is shown in Figure 6.
Revised: { c ll , c lu , c ul , c uu } is shown in Figure 6.

Original: The local bump φ c ( x ) evaluates to 1 for all x that lie within the convex hull of c , namely conv ( c ) , after which φ c ( x ) quickly decreases linearly to 0.
Revised: The local bump φ c ( x ) evaluates to 1 for all x that lie within the convex hull of c , namely conv ( c ) , after which φ c ( x ) quickly decreases linearly to 0.

Original: φ c has 1 + 2(2 d − 1)
Revised: φ c has 1 + 2(2 d − 1)

Original: + 2 d ReLUs and 1 + (cid:100) log 2 (2 d + 1) (cid:101)
Revised: + 2 d ReLUs and 1 + (cid:100) log 2 (2 d + 1) (cid:101)

Original: + 1 layers.
Revised: + 1 layers.

Original: 1 /M(cid:96) 1 /M 1 /M(cid:96) 1 /M
Revised: 1 /M(cid:96) 1 /M 1 /M(cid:96) 1 /M

Original: c ll c lu c ul c uu ( c lux ,c luy , 1) φ c
Revised: c ll c lu c ul c uu ( c lux ,c luy , 1) φ c

Original: By construction φ c ( x ) decreases to 0 before reaching the next neighboring grid points N ( conv ( c )) , where N ( x ) := { g ∈ G | || x − g || ∞ ≤ 1 M } \ { x } denotes the neighboring grid points of x and similarly for
Revised: By construction φ c ( x ) decreases to 0 before reaching the next neighboring grid points N ( conv ( c )) , where N ( x ) := { g ∈ G | || x − g || ∞ ≤ 1 M } \ { x } denotes the neighboring grid points of x and similarly for

Original: N ( U ) :
Revised: N ( U ) :

Original: = {N ( x ) | x ∈ U } \ U (Figure 4).
Revised: = {N ( x ) | x ∈ U } \ U (Figure 4).

Original: The set N ( conv ( c )) forms a hyperrectangle in G and is shown in Figure 6 using red squares.
Revised: The set N ( conv ( c )) forms a hyperrectangle in G and is shown in Figure 6 using red squares.

Original: Clearly conv ( c ) ⊆ conv ( N ( c )) .
Revised: Clearly conv ( c ) ⊆ conv ( N ( c )) .

Original: Next, we give bounds on the loss of precision for the interval-transformation φ (cid:93)c .
Revised: Next, we give bounds on the loss of precision for the interval-transformation φ (cid:93)c .

Original: We can show that interval analysis can (i) never produce intervals exceeding [0 , 1] and (ii) is precise if B does no intersect conv ( N ( c ))
Revised: We can show that interval analysis can (i) never produce intervals exceeding [0 , 1] and (ii) is precise if B does no intersect conv ( N ( c ))

Original: \ conv ( c ) .
Revised: \ conv ( c ) .

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: For all B ∈ B ( R m ) , it holds that φ (cid:93)c ( B ) ⊆
Revised: For all B ∈ B ( R m ) , it holds that φ (cid:93)c ( B ) ⊆

Original: [0 , 1] ∈ B and
Revised: [0 , 1] ∈ B and

Original: [Equation]
Revised: [Equation]

Original: The formal proof is given in Appendix A.
Revised: The formal proof is given in Appendix A.

Original: The next lemma shows, how a ReLU network n k can approximate the slice f k while simultaneously conﬁning the loss of analysis precision.
Revised: The next lemma shows, how a ReLU network n k can approximate the slice f k while simultaneously conﬁning the loss of analysis precision.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: Let Γ ⊂ R m be a closed box and let f : Γ → R be continuous.
Revised: Let Γ ⊂ R m be a closed box and let f : Γ → R be continuous.

Original: For all δ > 0 there exists a set of ReLU networks { n k } 0 ≤ k<N of size N ∈ N approximating the N -slicing of f , { f k } 0 ≤ k<N ( ξ k as in Deﬁnition 4.1) such that for all boxes B ∈ B (Γ)
Revised: For all δ > 0 there exists a set of ReLU networks { n k } 0 ≤ k<N of size N ∈ N approximating the N -slicing of f , { f k } 0 ≤ k<N ( ξ k as in Deﬁnition 4.1) such that for all boxes B ∈ B (Γ)

Original: [Equation]
Revised: [Equation]

Original: and n (cid:93) k
Revised: and n (cid:93) k

Original: ( B ) ⊆
Revised: ( B ) ⊆

Original: It is important to note that in Equation (2) we mean f and not f (cid:93) .
Revised: It is important to note that in Equation (2) we mean f and not f (cid:93) .

Original: The proof for Lemma 4.4 is given in Appendix A.
Revised: The proof for Lemma 4.4 is given in Appendix A.

Original: In the following, we discuss a proof sketch.
Revised: In the following, we discuss a proof sketch.

Original: Because Γ is compact and f is continuous, f is uniformly continuous by the Heine-Cantor Theorem.
Revised: Because Γ is compact and f is continuous, f is uniformly continuous by the Heine-Cantor Theorem.

Original: So we can pick a M ∈ N such that for all x,
Revised: So we can pick a M ∈ N such that for all x,

Original: y ∈ Γ satisfying ||
Revised: y ∈ Γ satisfying ||

Original: y − x || ∞ ≤ 1 M holds | f ( y ) − f ( x ) | ≤ δ 2 .
Revised: y − x || ∞ ≤ 1 M holds | f ( y ) − f ( x ) | ≤ δ 2 .

Original: We then choose the grid G =
Revised: We then choose the grid G =

Original: ( Z M ) m ⊆ R m .
Revised: ( Z M ) m ⊆ R m .

Original: Next, we construct for every slice k a set ∆ k of hyperrectangles on the grid G : if a box B ∈ B (Γ) fulﬁlls f ( B ) ≥ ξ k +1 + δ 2 , then we add a minimal enclosing hyperrectangle c ⊂ G such that B ⊆ conv ( c ) to ∆ k , where conv ( c ) denotes the convex hull of c .
Revised: Next, we construct for every slice k a set ∆ k of hyperrectangles on the grid G : if a box B ∈ B (Γ) fulﬁlls f ( B ) ≥ ξ k +1 + δ 2 , then we add a minimal enclosing hyperrectangle c ⊂ G such that B ⊆ conv ( c ) to ∆ k , where conv ( c ) denotes the convex hull of c .

Original: This implies, using uniform continuity of f and that the grid G is ﬁne enough, that f ( conv ( c )) ≥ ξ k +1 .
Revised: This implies, using uniform continuity of f and that the grid G is ﬁne enough, that f ( conv ( c )) ≥ ξ k +1 .

Original: Since there is only a ﬁnite number of possible hyperrectangles in G , the set ∆ k is clearly ﬁnite.
Revised: Since there is only a ﬁnite number of possible hyperrectangles in G , the set ∆ k is clearly ﬁnite.

Original: The network fulﬁlling
Revised: The network fulﬁlling

Original: Equation (2) is
Revised: Equation (2) is

Original: [Equation]
Revised: [Equation]

Original: where φ c is as in Deﬁnition 4.2.
Revised: where φ c is as in Deﬁnition 4.2.

Original: The n k are depicted in Figure 3c.
Revised: The n k are depicted in Figure 3c.

Original: Now, we see that Equation (2) holds by construction:
Revised: Now, we see that Equation (2) holds by construction:

Original: For all boxes B ∈ B (Γ) such that f ≥ ξ k +

Revised: For all boxes B ∈ B (Γ) such that f ≥ ξ k +

Original: +
Revised: +

Original: δon
Revised: δon

Original: B exists c (cid:

Revised: B exists c (cid:

Original: ) ∈
Revised: ) ∈

Original: ∆ k such that B ⊆ conv ( c (cid:48) ) which implies, using Lemma 4.3, that φ (cid:93)c (cid:48)
Revised: ∆ k such that B ⊆ conv ( c (cid:48) ) which implies, using Lemma 4.3, that φ (cid:93)c (cid:48)

Original: ( B ) =
Revised: ( B ) =

Original: [1 , 1] , hence
Revised: [1 , 1] , hence

Original: [Equation]
Revised: [Equation]

Original: Similarly, if f ( B ) ≤ ξ k − δ 2 holds, then it holds for all c ∈ ∆ k that B does not intersect N ( conv ( c )) .
Revised: Similarly, if f ( B ) ≤ ξ k − δ 2 holds, then it holds for all c ∈ ∆ k that B does not intersect N ( conv ( c )) .

Original: Indeed, if a c ∈ ∆ k would violate this, then by construction, f ( conv ( c )) ≥ ξ k +1 , contradicting f ( B ) ≤ ξ k − δ 2 .
Revised: Indeed, if a c ∈ ∆ k would violate this, then by construction, f ( conv ( c )) ≥ ξ k +1 , contradicting f ( B ) ≤ ξ k − δ 2 .

Original: Thus φ (cid:93)c ( B ) =
Revised: Thus φ (cid:93)c ( B ) =

Original: [0 , 0] , and hence n (cid:93) ( B ) =
Revised: [0 , 0] , and hence n (cid:93) ( B ) =

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: Let Γ ⊂ R m be a closed box and let f : Γ → R be continuous.
Revised: Let Γ ⊂ R m be a closed box and let f : Γ → R be continuous.

Original: Then for all δ > 0 , exists a ReLU network n such that for all B ∈ B (Γ)
Revised: Then for all δ > 0 , exists a ReLU network n such that for all B ∈ B (Γ)

Original: [Equation]
Revised: [Equation]

Original: where l := min f ( B ) and u := max f ( B ) .
Revised: where l := min f ( B ) and u := max f ( B ) .

Original: Proof.
Revised: Proof.

Original: Pick N such that the height of each slice is exactly δ 2 , if this is impossible choose a slightly smaller δ .
Revised: Pick N such that the height of each slice is exactly δ 2 , if this is impossible choose a slightly smaller δ .

Original: Let { n k } 0 ≤ k<
Revised: Let { n k } 0 ≤ k<

Original: N be a series of networks as in Lemma 4.4.
Revised: N be a series of networks as in Lemma 4.4.

Original: Recall that ξ 0 = min
Revised: Recall that ξ 0 = min

Original: f (Γ) .
Revised: f (Γ) .

Original: We deﬁne the ReLU network
Revised: We deﬁne the ReLU network

Original: [Equation]
Revised: [Equation]

Original: Let B ∈ B (Γ) .
Revised: Let B ∈ B (Γ) .

Original: Thus we have for all k
Revised: Thus we have for all k

Original: [Equation]
Revised: [Equation]

Revised: Lemma 4 .

Revised: [Equation]

Original: Let p, q ∈ { 0 , . . .
Revised: Let p, q ∈ { 0 , . . .

Original: , N − 1 } such that
Revised: , N − 1 } such that

Original: [Equation]
Revised: [Equation]

Original: as depicted in Figure 7.
Revised: depicted in Figure 7.

Original: Thus by Equation (4) for all k ∈ { 0 , . . .
Revised: Thus by Equation (4) for all k ∈ { 0 , . . .

Original: , p − 2 } it holds that n (cid:93)k ( B ) =
Revised: , p − 2 } it holds that n (cid:93)k ( B ) =

Original: [1 , 1] and similarly, by Equation (5) for all k ∈ { q + 2 , . . .
Revised: [1 , 1] and similarly, by Equation (5) for all k ∈ { q + 2 , . . .

Original: , N − 1 } it holds that n (cid:93)k ( B ) =
Revised: , N − 1 } it holds that n (cid:93)k ( B ) =

Original: Plugging this into Equation (3) after splitting the sum into three parts leaves us with n (cid:93) ( B ) =
Revised: Plugging this into Equation (3) after splitting the sum into three parts leaves us with n (cid:93) ( B ) =

Original: ξ 0 + δ
Revised: ξ 0 + δp

Original: [Equation]
Revised: [Equation]

Revised: (cid:88) k

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: B

Revised: [Equation]

Revised: + δ

Revised: [Equation]

Original: Applying the standard rules for interval analysis
Revised: Applying the standard rules for interval analysis

Original: [Equation]
Revised: [Equation]

Original: leads to
Revised: leads to

Original: [Equation]
Revised: [Equation]

Original: where we used in the last step
Revised: where we used in the last step

Original: [Equation]
Revised: [Equation]

Original: that ξ 0 + k δ 2 = ξ k .
Revised: that ξ 0 + k δ 2 = ξ k .

Original: For all terms in the sum except the terms corresponding to the 3 highest and lowest k we get
Revised: For all terms in the sum except the terms corresponding to the 3 highest and lowest k we get

Original: [Equation]
Revised: [Equation]

Original: Indeed, from Equation (6) we know that there is x ∈ B such that f ( x ) ≤ ξ p +1 =
Revised: Indeed, from Equation (6) we know that there is x ∈ B such that f ( x ) ≤ ξ p +1 =

Original: ξ p +2 − δ 2 , thus by
Revised: ξ p +2 − δ 2 , thus by

Original: Lemma 4.4 n (cid:93)k ([ x, x ]) =
Revised: Lemma 4.4 n (cid:93)k ([ x, x ]) =

Original: [0 , 0] for all p + 2 ≤ k ≤ q − 2 .
Revised: [0 , 0] for all p + 2 ≤ k ≤ q − 2 .

Original: Similarly, from Equation (7) we know, that there is x (cid:48)
Revised: Similarly, from Equation (7) we know, that there is x (cid:48)

Original: ∈ B such that f ( x ) ≥ ξ q = ξ q − 1 + δ 2 , thus by Lemma 4.4 n (cid:93)k ([ x (cid:48) , x (cid:48) ]) =
Revised: ∈ B such that f ( x ) ≥ ξ q = ξ q − 1 + δ 2 , thus by Lemma 4.4 n (cid:93)k ([ x (cid:48) , x (cid:48) ]) =

Original: [1 , 1] for all p + 2 ≤ k ≤ q − 2 .
Revised: [1 , 1] for all p + 2 ≤ k ≤ q − 2 .

Original: So n (cid:93)k ( B ) is at least [0 , 1] , and by Lemma 4.4 also at most [0 , 1] .
Revised: So n (cid:93)k ( B ) is at least [0 , 1] , and by Lemma 4.4 also at most [0 , 1] .

Original: This leads to n
Revised: This leads to n

Original: [Equation]
Revised: [Equation]

Original: We know further, that if p + 3 ≤ q , than there is an x ∈ B such that f ( x ) ≥ ξ p +3 =
Revised: We know further, that if p + 3 ≤ q , than there is an x ∈ B such that f ( x ) ≥ ξ p +3 =

Original: ξ p +2 + δ 2 , hence similar as before n (cid:93)p +1 ([ x, x ]) =
Revised: ξ p +2 + δ 2 , hence similar as before n (cid:93)p +1 ([ x, x ]) =

Original: [1 , 1] and similarly n (cid:93)p ([ x, x ]) =
Revised: [1 , 1] and similarly n (cid:93)p ([ x, x ]) =

Original: [1 , 1] and n (cid:93) ([ x, x ]) =
Revised: [1 , 1] and n (cid:93) ([ x, x ]) =

Original: So we know, that δ 2 (cid:80) p +1 k
Revised: So we know, that δ 2 (cid:80) p +1 k

Original: = p − 1 n (cid:93)k ( B ) includes at least [3 δ 2 , 3 δ 2 ] and at the most [0 , 3 δ 2 ] .
Revised: = p − 1 n (cid:93)k ( B ) includes at least [3 δ 2 , 3 δ 2 ] and at the most [0 , 3 δ 2 ] .

Original: Similarly, there exists
Revised: Similarly, there exists

Original: an x (cid:48) ∈
Revised: an x (cid:48) ∈

Original: B such that n (cid:93)q − 1 ([ x (cid:48) , x (cid:48) ])
Revised: B such that n (cid:93)q − 1 ([ x (cid:48) , x (cid:48) ])

Original: =
Revised: =

Original: [0 , 0] , n (cid:93)q ([ x (cid:48) , x (cid:48) ]) =
Revised: [0 , 0] , n (cid:93)q ([ x (cid:48) , x (cid:48) ]) =

Original: [0 , 0] and n (cid:93)q +1 ([ x (cid:48) , x (cid:48) ]) [Equation]
Revised: [0 , 0] and n (cid:93)q +1 ([ x (cid:48) , x (cid:48) ]) =

Original: This leaves us with
Revised: This leaves us with

Original: [Equation]
Revised: [Equation]

Original: If p + 3 > q the lower bound we want to prove becomes vacuous and only the upper one needs to be proven.
Revised: If p + 3 > q the lower bound we want to prove becomes vacuous and only the upper one needs to be proven.

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: where l := min f ( B ) and u := max f ( B ) .
Revised: where l := min f ( B ) and u := max f ( B ) .

Original: Theorem 4.6 (Universal Interval-Provable Approximation) .
Revised: Theorem 4.6 (Universal Interval-Provable Approximation) .

Original: Let Γ ⊂
Revised: Let Γ ⊂

Original: R m be compact and f : Γ → R d be continuous.
Revised: R m be compact and f : Γ → R d be continuous.

Original: For all δ ∈ R m ≥ 0 exists a ReLU network n such that for all B ∈ B (Γ)
Revised: For all δ ∈ R m ≥ 0 exists a ReLU network n such that for all B ∈ B (Γ)

Original: [Equation]
Revised: [Equation]

Original: where l, u ∈ R m such that l k :
Revised: where l, u ∈ R m such that l k :

Original: = min f ( B ) k and u k :=
Revised: = min f ( B ) k and u k :=

Original: max f ( B ) k for all k .
Revised: max f ( B ) k for all k .

Original: Proof.
Revised: Proof.

Original: This is a direct consequence of using Theorem 4.5 and the Tietze extension theorem to produce a neural network for each dimension d of the codomain of f .
Revised: This is a direct consequence of using Theorem 4.5 and the Tietze extension theorem to produce a neural network for each dimension d of the codomain of f .

Original: Note that Theorem 1.1 is a special case of Theorem 4.6 with d = 1 to simplify presentation.
Revised: Note that Theorem 1.1 is a special case of Theorem 4.6 with d = 1 to simplify presentation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proved that for all real valued continuous functions f on compact sets, there exists a ReLU network n approximating f arbitrarily well with the interval abstraction.
Revised: We proved that for all real valued continuous functions f on compact sets, there exists a ReLU network n approximating f arbitrarily well with the interval abstraction.

Original: This means that for arbitrary input sets, analysis using the interval relaxation yields an over-approximation arbitrarily close to the smallest interval containing all possible outputs.
Revised: This means that for arbitrary input sets, analysis using the interval relaxation yields an over-approximation arbitrarily close to the smallest interval containing all possible outputs.

Original: Our theorem afﬁrmatively answers the open question, whether the Universal Approximation Theorem generalizes to Interval analysis.
Revised: Our theorem afﬁrmatively answers the open question, whether the Universal Approximation Theorem generalizes to Interval analysis.

Original: Our results address the question of whether the interval abstraction is expressive enough to analyse networks approximating interesting functions f .
Revised: Our results address the question of whether the interval abstraction is expressive enough to analyse networks approximating interesting functions f .

Original: This is of practical importance because interval analysis is the most scalable non-trivial analysis.
Revised: This is of practical importance because interval analysis is the most scalable non-trivial analysis.

Original: A P ROOFS FOR THE U NIVERSAL I NTERVAL -C ERTIFIED A PPROXIMATION
Revised: A P ROOFS FOR THE U NIVERSAL I NTERVAL -C ERTIFIED A PPROXIMATION

Original: Lemma A.1 (Monotonicity) .
Revised: Lemma A.1 (Monotonicity) .

Original: The operations + , − are monotone, that is for all [ a 1 , b 1 ] , [ a 2 , b 2 ] , [ c 1 , d 1 ] , [ c 2 , d 2 ] ∈ B ( R ) such that [ a 1 , b 1 ] ⊆
Revised: The operations + , − are monotone, that is for all [ a 1 , b 1 ] , [ a 2 , b 2 ] , [ c 1 , d 1 ] , [ c 2 , d 2 ] ∈ B ( R ) such that [ a 1 , b 1 ] ⊆

Original: [ a 2 , b 2 ] and [ c 1 , d 2 ] ⊆
Revised: [ a 2 , b 2 ] and [ c 1 , d 2 ] ⊆

Original: [ c 2 , d 2 ] holds
Revised: [ c 2 , d 2 ] holds

Original: [Equation]
Revised: [Equation]

Original: Further the operation ∗ and R are monotone, that is for all [ a [Equation] b ] , [ c, d ] ∈ B ( R ) and for all λ ∈ R ≥such that [ a, b ] ⊆
Revised: Further the operation ∗ and R are monotone, that is for all [ a, b ] , [ c, d ] ∈ B ( R ) and for all λ ∈ R ≥such that [ a, b ] ⊆

Original: [ c, d ] holds
Revised: [ c, d ] holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition A.2 ( N -slicing) .
Revised: Deﬁnition A.2 ( N -slicing) .

Original: Let Γ ⊂
Revised: Let Γ ⊂

Original: R m be a compact m -dimensional box and let f : Γ → R be continuous.
Revised: R m be a compact m -dimensional box and let f : Γ → R be continuous.

Original: The N -slicing of f is a set of functions { f k } 0 ≤ k ≤ N − 1 deﬁned by
Revised: The N -slicing of f is a set of functions { f k } 0 ≤ k ≤ N − 1 deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where ξ k := kN ( ξ max − ξ min ) , k ∈ { 0 , . .
Revised: where ξ k := kN ( ξ max − ξ min ) , k ∈ { 0 , . .

Original: . , N } , ξ min :=
Revised: . , N } , ξ min :=

Original: min f (Γ) and ξ max := max f (Γ) .
Revised: min f (Γ) and ξ max := max f (Γ) .

Original: Lemma A.3 ( N -slicing) .
Revised: Lemma A.3 ( N -slicing) .

Original: Let { f k } 0 ≤ k ≤ N − 1 be the N -slicing of f .
Revised: Let { f k } 0 ≤ k ≤ N − 1 be the N -slicing of f .

Original: Then for all x ∈ Γ we have f ( x ) :
Revised: Then for all x ∈ Γ we have f ( x ) :

Original: =
Revised: =

Original: ξ 

Revised: ξ

Original: + (cid:80) N − 1 k =

Revised: + (cid:80) N − 1 k =

Original: f k ( x ) .
Revised: f k ( x ) .

Original: Proof.
Revised: Proof.

Original: Pick x ∈ Γ and let l ∈ { 0 , . .
Revised: Pick x ∈ Γ and let l ∈ { 0 , . .

Original: , N − 1 } such that ξ l ≤ f ( x ) ≤ ξ l +1 .
Revised: , N − 1 } such that ξ l ≤ f ( x ) ≤ ξ l +1 .

Original: Then ξ 0 +
Revised: Then ξ 0 +

Original: N − 1 (cid:88)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition A.4 (clipping) .
Revised: Deﬁnition A.4 (clipping) .

Original: Let a, b ∈ R , a < b .
Revised: Let a, b ∈ R , a < b .

Original: We deﬁne the clipping function R [ ∗ ,b ] : R → R by
Revised: We deﬁne the clipping function R [ ∗ ,b ] : R → R by

Original: R [ ∗ ,b ] ( x ) :=
Revised: R [ ∗ ,b ] ( x ) :=

Original: b − R ( b − x ) .
Revised: b − R ( b − x ) .

Original: Lemma A.5 (clipping) .
Revised: Lemma A.5 (clipping) .

Original: The function R [ ∗ ,b ] sends all x ≤ b to x , and all x > b to b .
Revised: The function R [ ∗ ,b ] sends all x ≤ b to x , and all x > b to b .

Original: Further,
Revised: Further,

Original: R (cid:93) [ ∗ ,b ] ([ a (cid:48) , b (cid:48) ]) =
Revised: R (cid:93) [ ∗ ,b ] ([ a (cid:48) , b (cid:48) ]) =

Original: [ R [ ∗ ,b ] ( a (cid:48) ) , R [ ∗ ,b ] ( b (cid:48) )] .
Revised: [ R [ ∗ ,b ] ( a (cid:48) ) , R [ ∗ ,b ] ( b (cid:48) )] .

Original: Proof.
Revised: Proof.

Original: We show the proof for R [ a,b ] , the proof for R [ ∗ ,b ] is similar.
Revised: We show the proof for R [ a,b ] , the proof for R [ ∗ ,b ] is similar.

Original: [Equation]
Revised: [Equation]

Original: Next,
Revised: Next,

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition A.6 (nmin) .
Revised: Deﬁnition A.6 (nmin) .

Original: We deﬁne the ReLU network nmin : R 2 → R by
Revised: We deﬁne the ReLU network nmin : R 2 → R by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:19) .

Original: Lemma A.7 (nmin) .
Revised: Lemma A.7 (nmin) .

Original: Let x, y ∈ R , then nmin ( x, y )
Revised: Let x, y ∈ R , then nmin ( x, y )

Original: =
Revised: =

Original: min( x, y ) .
Revised: min( x, y ) .

Original: Proof.
Revised: Proof.

Original: Because nmin is symmetric in its arguments, we assume w.o.l.g. x ≥ y .
Revised: Because nmin is symmetric in its arguments, we assume w.o.l.g. x ≥ y .

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: If x + y ≥ 0 , then
Revised: If x + y ≥ 0 , then

Original: [Equation]
Revised: [Equation]

Original: If x + y < 0 , then
Revised: If x + y < 0 , then

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition A.8 (nmin N ) .
Revised: Deﬁnition A.8 (nmin N ) .

Original: For all N ∈ N ≥ 1 , we deﬁne a ReLU network nmin N deﬁned by 
Revised: For all N ∈ N ≥ 1 , we deﬁne a ReLU network nmin N deﬁned by nmin

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: nmin N ( x 1 , . . .
Revised: nmin N ( x 1 , . . .

Original: , x N ) :
Revised: , x N ) :

Original: = nmin ( nmin
Revised: = nmin ( nmin

Original: [Equation]
Revised: [Equation]

Original: N/
Revised: N/

Original: [Equation]
Revised: [Equation]

Revised: ( x 1 , . . .

Revised: , x

Original: [Equation]
Revised: [Equation]

Original: , x
Revised: N

Original: [Equation]
Revised: [Equation]

Original: ) , nmin
Revised: ) , nmin

Original: [Equation]
Revised: [Equation]

Original: +
Revised: N

Original: [Equation]
Revised: [Equation]

Original:  ( x (cid:100) N
Revised: +1 ( x (cid:100) N

Original: [Equation]
Revised: [Equation]

Original: (cid:101) +1 , . . .
Revised: (cid:101) +1 , . . .

Original: , x N )) .
Revised: , x N )) .

Original: Lemma A.9.
Revised: Lemma A.9.

Original: Let [ a, b ] , [ c, d ] ∈ B ( R ) .
Revised: Let [ a, b ] , [ c, d ] ∈ B ( R ) .

Original: Then nmin (cid:93) ([ a, b ] , [ c, d ]) =
Revised: Then nmin (cid:93) ([ a, b ] , [ c, d ]) =

Original: nmin (cid:93) ([ c, d ] , [ a, b ]) and nmin (cid:93) ([ a, b ] , [ c, d ]) =
Revised: nmin (cid:93) ([ c, d ] , [ a, b ]) and nmin (cid:93) ([ a, b ] , [ c, d ]) =

Original:   [ c + a − b 2 , d + b − a 2 ] if d ≤ a [ a + c − d 2 , b + d − c 2 ] if a ≤ d and b < c
Revised:   [ c + a − b 2 , d + b − a 2 ] if d ≤ a [ a + c − d 2 , b + d − c 2 ] if a ≤ d and b < c

Original: [ a + c − b + d 2 , b + d 2 ] if a ≤ d and b ≥ c
Revised: [ a + c − b + d 2 , b + d 2 ] if a ≤ d and b ≥ c

Original: Proof.
Revised: Proof.

Original: The symmetry on abstract elements is immediate.
Revised: The symmetry on abstract elements is immediate.

Original: In the following, we omit some of (cid:93) to improve readability. 
Revised: In the following, we omit some of (cid:93) to improve readability. [Equation] [Equation] a

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: d )

Revised: [Equation]

Revised: R

Revised: [Equation]

Revised: b

Revised: [Equation]

Revised: c )]

Revised: [Equation]

Revised: [ R

Revised: [Equation]

Revised: c

Revised: [Equation]

Revised: b )

Revised: [Equation]

Revised: R

Revised: [Equation]

Revised: d

Revised: [Equation]

Revised: a )

Revised: [Equation]

Revised: [Equation]

Revised: d )]

Revised: [Equation]

Revised: [Equation]

Revised: R

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: c )

Revised: [Equation]

Revised: − R

Revised: [Equation]

Revised: d )]

Revised: [Equation]

Revised: [Equation]

Revised: c

Original:  Claim: R ( a + c ) − R ( − a − c )
Revised: [Equation] − [Equation] Claim: R ( a + c ) − R ( − a − c )

Original: = a + c .
Revised: = a + c .

Original: If a + c > 0 then − a − c < 0 thus the claim in this case.
Revised: If a + c > 0 then − a − c < 0 thus the claim in this case.

Original: Indeed:
Revised: Indeed:

Original: If a + c ≤ 0 then − a − c ≥ 0 thus R ( a + c ) − R ( − a − c ) = − R ( − a − c ) = − ( − a − c ) = a + c .
Revised: If a + c ≤ 0 then − a − c ≥ 0 thus R ( a + c ) − R ( − a − c ) = − R ( − a − c ) = − ( − a − c ) = a + c .

Original: Similarly R ( b + d ) − R ( − b −
Revised: Similarly R ( b + d ) − R ( − b −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: b + d .
Revised: b + d .

Original: So the expression simpliﬁes to
Revised: So the expression simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: We proceed by case distinction:
Revised: We proceed by case distinction:

Original: Case 1: b − c ≤ 0 : Then a ≤ b≤ c
Revised: [Equation] 1: b − c ≤ 0 : Then a ≤ b

Original: [Equation]
Revised: [Equation]

Original: d :

Original: [Equation]
Revised: [Equation]

Original: Case 2: a − d ≥ 0 :
Revised: Case 2: a − d ≥ 0 :

Original: Then c ≤ d ≤ a ≤ b .
Revised: Then c ≤ d ≤ a ≤ b .

Original: By symmetry of nmin equivalent to Case 1.
Revised: By symmetry of nmin equivalent to Case 1.

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Case : a − d < 0 and b − c > 0 :
Revised: Case 3: a − d < 0 and b − c > 0 :

Original: [Equation]
Revised: [Equation]

Revised: a, b ] ,

Revised: [Equation]

Revised: c, d ]

Original: [Equation]
Revised: [Equation]

Original: Thus we have nmin (cid:93) ([ a,
Revised: Thus we have nmin (cid:93) ([ a,

Original: [Equation]
Revised: [Equation]

Original: ] , [ c,
Revised: ] , [ c,

Original: [Equation]
Revised: [Equation]

Original: ]) =
Revised: ]) =

Original: 

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [ a
Revised: [ a

Original: [Equation]
Revised: [Equation]

Original: c −
Revised: c −

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ] if a < d and b > c
Revised: ] if a < d and b > c

Original: Deﬁnition A.10 (neighboring grid points) .
Revised: Deﬁnition A.10 (neighboring grid points) .

Original: Let G be as above.
Revised: Let G be as above.

Original: We deﬁne the set of neighboring grid points of x ∈ Γ by
Revised: We deﬁne the set of neighboring grid points of x ∈ Γ by

Original: [Equation]
Revised: [Equation]

Original: For U ⊂ R m , we deﬁne N ( U ) :
Revised: For U ⊂ R m , we deﬁne N ( U ) :

Original: = {N ( x ) | x ∈ U } \ U .
Revised: = {N ( x ) | x ∈ U } \ U .

Original: Deﬁnition A.11 (local bump) .
Revised: Deﬁnition A.11 (local bump) .

Original: Let M ∈ N , G :=
Revised: Let M ∈ N , G :=

Original: ( Z M ) m , (cid:96) = 2 (cid:100) log 2 2 m (cid:101) +1 and let c
Revised: ( Z M ) m , (cid:96) = 2 (cid:100) log 2 2 m (cid:101) +1 and let c

Original: = { i l 1 M , i u 1 M } × · · · × { i lm M , i um M } ⊆ G .
Revised: = { i l 1 M , i u 1 M } × · · · × { i lm M , i um M } ⊆ G .

Original: We deﬁne a ReLU neural network φ c : R m → [0 , 1] w.r.t.
Revised: We deﬁne a ReLU neural network φ c : R m → [0 , 1] w.r.t.

Original: the grid G by
Revised: the grid G by

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: Lemma A.12.
Revised: Lemma A.12.

Original: It holds:
Revised: It holds:

Original: [Equation]
Revised: [Equation]

Original: + 1 } (cid:17) otherwise 
Revised: 1 } (cid:17) otherwise .

Original: Proof.
Revised: Proof.

Original: By case distinction:
Revised: By case distinction:

Original: • Case x / ∈ N ( c ) .
Revised: • Case x / ∈ N ( c ) .

Original: Then there exists k , such that either x k < i lk − 1 M or x k > i uk +1 M .
Revised: Then there exists k , such that either x k < i lk − 1 M or x k > i uk +1 M .

Original: Then M(cid:96)
Revised: Then M(cid:96)

Original: ( x k − i lk M ) + 1 or M(cid:96)
Revised: ( x k − i lk M ) + 1 or M(cid:96)

Original: ( i uk M − x k ) + 1 is less or equal to 0.
Revised: ( i uk M − x k ) + 1 is less or equal to 0.

Original: Hence φ c ( x ) = 0 .
Revised: Hence φ c ( x ) = 0 .

Original: • Case x ∈ conv ( c ) .
Revised: • Case x ∈ conv ( c ) .

Original: Then for all k holds i lk M ≤ x k ≤ i uk M .
Revised: Then for all k holds i lk M ≤ x k ≤ i uk M .

Original: Thus M(cid:96)
Revised: Thus M(cid:96)

Original: ( x k − i lk M ) + 1 ≥ 1 and M(cid:96)
Revised: ( x k − i lk M ) + 1 ≥ 1 and M(cid:96)

Original: ( i uk M − x k ) + 1 ≥ 1 for all
Revised: ( i uk M − x k ) + 1 ≥ 1 for all

Original: k
Revised: k

Original: Hence φ c ( x ) = 1 .
Revised: Hence φ c ( x ) = 1 .

Original: where α ≥ 1 .
Revised: where α ≥ 1 .

Original: • Case otherwise: For all x exists a k such that M(cid:96)
Revised: • Case otherwise: For all x exists a k such that M(cid:96)

Original: ( x k − i lk M ) + 1 or M(cid:96)
Revised: ( x k − i lk M ) + 1 or M(cid:96)

Original: ( i uk M − x k ) + 1 is smaller or equal to all other arguments of the function min and smaller or equal to 1 .
Revised: ( i uk M − x k ) + 1 is smaller or equal to all other arguments of the function min and smaller or equal to 1 .

Original: If the smallest element is smaller than 0, then φ c ( x ) will evaluate to 0, otherwise it will evaluate to M(cid:96)
Revised: If the smallest element is smaller than 0, then φ c ( x ) will evaluate to 0, otherwise it will evaluate to M(cid:96)

Original: ( x k − i lk M ) + 1 or M(cid:96)
Revised: ( x k − i lk M ) + 1 or M(cid:96)

Original: ( i uk M − x k ) + 1 .
Revised: ( i uk M − x k ) + 1 .

Original: Thus we can just drop R and R [ ∗ , 1] from the equations and take the minimum also over 0:
Revised: Thus we can just drop R and R [ ∗ , 1] from the equations and take the minimum also over 0:

Original: [Equation]
Revised: [Equation]

Original: Lemma A.13.
Revised: Lemma A.13.

Original: Let [ u 1 , 1] , . . .
Revised: Let [ u 1 , 1] , . . .

Original: , [ u N , 1] be abstract elements of the Interval Domain B .
Revised: , [ u N , 1] be abstract elements of the Interval Domain B .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By induction.
Revised: By induction.

Original: Base case: Let N = 1 .
Revised: Base case: Let N = 1 .

Original: Then nmin (cid:93) 1 ([ u 1 , 1]) =
Revised: Then nmin (cid:93) 1 ([ u 1 , 1]) =

Original: [ u 1 , 1] .
Revised: [ u 1 , 1] .

Original: Let N = 2 .
Revised: Let N = 2 .

Original: Then nmin (cid:93) 2 ([ u 1 , 1] , [ u 2 , 1]) =
Revised: Then nmin (cid:93) 2 ([ u 1 , 1] , [ u 2 , 1]) =

Original: [ u 1 + u 2 − 1 , 1] .
Revised: [ u 1 + u 2 − 1 , 1] .

Original: Induction hypothesis: The property holds for N (cid:48)
Revised: Induction hypothesis: The property holds for N (cid:48)

Original: s.t
Revised: s.t

Original: [Equation]
Revised: [Equation]

Original: 0 < N [Equation] ≤ N − 1 .
Revised: 0 < N (cid:48) ≤ N − 1 .

Original: Induction step: Then it also holds for N :
Revised: Induction step: Then it also holds for N :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Lemma A.14.
Revised: Lemma A.14.

Original: Let [ a, b ] , [ u, 1] ∈ B
Revised: Let [ a, b ] , [ u, 1] ∈ B

Original: ( R ≤ 1 ) .
Revised: ( R ≤ 1 ) .

Original: Then nmin
Revised: Then nmin

Original: [Equation]
Revised: [Equation]

Original:  a,
Revised: [ a,

Original: [Equation]
Revised: [Equation]

Original: ] , [ u, 1]
Revised: ] , [ u, 1]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +12 ]
Revised: +12 ]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: If b ≤ u then b + 1 − u 2 ≤ b + 1 − b 2 = b +12 . If u ≤ b then a + u − b +12 ≥ a + u − u +12 = a + u − 12 .
Revised: If b ≤ u then b + 1 − u 2 ≤ b + 1 − b 2 = b +12 . If u ≤ b then a + u − b +12 ≥ a + u − u +12 = a + u − 12 .

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: Lemma A.15.
Revised: Lemma A.15.

Original: Let N ∈ N ≥ 2 , let [ u 1 , 1] , . .
Revised: Let N ∈ N ≥ 2 , let [ u 1 , 1] , . .

Original: , [ u N − 1 , 1] , [ u N , d ] ∈ B ( R ) s.t. b ≤ 1 be abstract elements of the Interval Domain B .
Revised: , [ u N − 1 , 1] , [ u N , d ] ∈ B ( R ) s.t. b ≤ 1 be abstract elements of the Interval Domain B .

Original: Furthermore, let H ( x ) :
Revised: Furthermore, let H ( x ) :

Original: = 1+ x 2 .
Revised: = 1+ x 2 .

Original: Then there exists a u ∈ R s.t.
Revised: Then there exists a u ∈ R s.t.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By induction: Let N = 2 :
Revised: By induction: Let N = 2 :

Original: [Equation]
Revised: [Equation]

Original: Let N = 3 :
Revised: Let N = 3 :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: So nmin (cid:93) 3 ([ u 3 , d ] , [ u 1 , 1] , [ u 2 , 1]) is always included in [ u 3 + u 1 + u 2 − 2, H 2 ( d )] .
Revised: So nmin (cid:93) 3 ([ u 3 , d ] , [ u 1 , 1] , [ u 2 , 1]) is always included in [ u 3 + u 1 + u 2 − 2, H 2 ( d )] .

Original: Induction hypothesis: The statement holds for all 2 ≤ N (cid:48) ≤ N − 1 .
Revised: Induction hypothesis: The statement holds for all 2 ≤ N (cid:48) ≤ N − 1 .

Original: Induction step: Then the property holds also for N :
Revised: Induction step: Then the property holds also for N :

Original: [Equation]
Revised: [Equation]

Original: (cid:101) +1 ( d )] and similarly for other orderings of the arguments.
Revised: +1 ( d )] and similarly for other orderings of the arguments.

Original: Lemma A.16.
Revised: Lemma A.16.

Original: Let H ( x ) :
Revised: Let H ( x ) :

Original: = 1+ x 2 .
Revised: = 1+ x 2 .

Original: For all N ∈ N > 0 , we have that d ≤ 1 − 2 N implies H N ( d ) ≤ 0 .
Revised: For all N ∈ N > 0 , we have that d ≤ 1 − 2 N implies H N ( d ) ≤ 0 .

Original: Proof.
Revised: Proof.

Original: By induction.
Revised: By induction.

Original: N = 1 : Then H (1 − 2)
Revised: N = 1 : Then H (1 − 2)

Original: = 1+1 − 2=
Revised: = 1+1 − 2=

Original: Induction hypothesis.
Revised: Induction hypothesis.

Original: The statement holds for all N (cid:48) such that 0 < N (cid:48) ≤ N .
Revised: The statement holds for all N (cid:48) such that 0 < N (cid:48) ≤ N .

Original: Induction step: N +1 :
Revised: Induction step: N +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma A.17.
Revised: Lemma A.17.

Original: For all boxes B ∈ B ( R m ) , we have
Revised: For all boxes B ∈ B ( R m ) , we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, φ (cid:93)c ( B ) ⊆
Revised: Furthermore, φ (cid:93)c ( B ) ⊆

Original: Proof.
Revised: Proof.

Original: Let φ c be a local bump and let B =
Revised: Let φ c be a local bump and let B =

Original: [ a, b ] ∈ B ( R m ) .
Revised: [ a, b ] ∈ B ( R m ) .

Original: Let [ r 1 k , s 1 k ] , [ r 2 k , s 2 k ] ∈ B ( R ) such that
Revised: Let [ r 1 k , s 1 k ] , [ r 2 k , s 2 k ] ∈ B ( R ) such that

Original: M(cid:96)
Revised: M(cid:96)

Original: ([ a k , b k ] − i lk M ) + 

Revised: ([ a k , b k ] − i lk M ) +

Original: =
Revised: =

Original: [ r 1 k , s 1 k ] and M(cid:96)
Revised: [ r 1 k , s 1 k ] and M(cid:96)

Original: ( i uk M − [ a k , b k ]) + 1{+
+} =
Revised: ( i uk M − [ a k , b k ]) + {+
+} =

Original:  r 2 k , s 2 k ] .
Revised: [ r 2 k , s 2 k ] .

Original: If [ a, b ] ⊆ conv ( c ) : Then 1 ≤ r 1 k and 1 ≤ r 2 k for all k ∈ { 1 , . . . , m } .
Revised: If [ a, b ] ⊆ conv ( c ) : Then 1 ≤ r 1 k and 1 ≤ r 2 k for all k ∈ { 1 , . . . , m } .

Original: Thus φ (cid:93)c ([ a, b ]) =
Revised: Thus φ (cid:93)c ([ a, b ]) =

Original: R (cid:93) ( nmin (cid:93)
Revised: R (cid:93) ( nmin (cid:93)

Original: [Equation]
Revised: [Equation]

Original:  (cid:93) [Equation] 1] ([ r p k , s p k ]) }
Revised: { R (cid:93) [ ∗ , 1] ([ r p k , s p k ]) }

Original: ( p,k ) 
Revised: ( p,k ) ∈

Original: [Equation]
Revised: [Equation]

Original: • If [ a, b ] ⊆ Γ \ conv ( N ( c )) : Then there exists a ( p (cid:48) , k (cid:48) )
Revised: • If [ a, b ] ⊆ Γ \ conv ( N ( c )) : Then there exists a ( p (cid:48) , k (cid:48) )

Original: ∈ { 1 , 2 } × { 1 , . .
Revised: ∈ { 1 , 2 } × { 1 , . .

Original: , m } such that s p [Equation]
Revised: , m } such that s p (cid:48)

Original: k (cid:48) ≤ 1 − 2 (cid:100) log 2 N (cid:101) +1 .
Revised: k (cid:48) ≤ 1 − 2 (cid:100) log 2 N (cid:101) +1 .

Original: Using Lemma A.16 and Lemma A.15, we now that there exists a
Revised: Using Lemma A.16 and Lemma A.15, we now that there exists a

Original: u ∈ R
Revised: u ∈ R

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t.

Original: [Equation]
Revised: [Equation]

Original: = [0 , 0]
Revised: [Equation]

Original: For any [ a, b ] ∈ B (Γ) we have φ (cid:93)c ([ a, b ]) ⊆
Revised: For any [ a, b ] ∈ B (Γ) we have φ (cid:93)c ([ a, b ]) ⊆

Original: [0 , 1] by construction.
Revised: [0 , 1] by construction.

Original: Lemma A.18.
Revised: Lemma A.18.

Original: Let Γ ⊂ R m be a closed box and let f : Γ → R be continuous.
Revised: Let Γ ⊂ R m be a closed box and let f : Γ → R be continuous.

Original: For all δ >exists a set of ReLU networks { n k } 0 ≤ k ≤ N − 1 of size N ∈ N approximating the N -slicing of f , { f k } 0 ≤ k ≤ N − 1 ( ξ k as in Deﬁnition A.2) such that for all boxes B ∈ B (Γ)
Revised: For all δ >exists a set of ReLU networks { n k } 0 ≤ k ≤ N − 1 of size N ∈ N approximating the N -slicing of f , { f k } 0 ≤ k ≤ N − 1 ( ξ k as in Deﬁnition A.2) such that for all boxes B ∈ B (Γ)

Original: [Equation]
Revised: [Equation]

Original: and n (cid:93)k ( B ) ⊆
Revised: and n (cid:93)k ( B ) ⊆

Original: Proof.
Revised: Proof.

Original: Let N ∈ N such that N ≥ 2 ξ max − ξ min δ where ξ min :=
Revised: Let N ∈ N such that N ≥ 2 ξ max − ξ min δ where ξ min :=

Original: min f (Γ) and ξ max := max f (Γ) .
Revised: min f (Γ) and ξ max := max f (Γ) .

Original: For simplicity we assume Γ = [0 , 1] m .
Revised: For simplicity we assume Γ = [0 , 1] m .

Original: Using the Heine-Cantor theorem, we get that f is uniformly continuous, thus there exists a δ (cid:48) > 0 such that ∀ x, y ∈ Γ .
Revised: Using the Heine-Cantor theorem, we get that f is uniformly continuous, thus there exists a δ (cid:48) > 0 such that ∀ x, y ∈ Γ .

Original: ||
Revised: ||

Original: y − x || ∞ <
Revised: y − x || ∞ <

Original: δ (cid:48)
Revised: δ (cid:48)

Original: ⇒ ||
Revised: ⇒ ||

Original: f ( y )
Revised: f ( y )

Original: − f ( x ) || <
Revised: − f ( x ) || <

Original: δ 2 .
Revised: δ 2 .

Original: Further, let M ∈ N such that M ≥ 1 δ (cid:48) and let G be the grid deﬁned by G :=
Revised: Further, let M ∈ N such that M ≥ 1 δ (cid:48) and let G be the grid deﬁned by G :=

Original: ( Z M ) m ⊆
Revised: ( Z M ) m ⊆

Original: R m .
Revised: R m .

Original: Let C ( B ) be the set of corner points of the closest hyperrectangle in G conﬁning B ∈ B (Γ) .
Revised: Let C ( B ) be the set of corner points of the closest hyperrectangle in G conﬁning B ∈ B (Γ) .

Original: We construct the set∆ k :=
Revised: We construct the set

Original: { C ( B ) | B ∈ B

Original: [Equation]
Revised: [Equation]

Original: [-Γ) : f ( B ) ≥ ξ k +
-]
Revised: [-Γ) : f ( B ) ≥ ξ k +
-]

Original: + δ 2 } .

Original: We claim that { n k } 0 ≤ k ≤ N − 1 deﬁned by
Revised: We claim that { n k } 0 ≤ k ≤ N − 1 deﬁned by

Original: [Equation]
Revised: [Equation]

Original: satisﬁes the condition.
Revised: satisﬁes the condition.

Original: Case 1: Let B ∈ B (Γ) such that f ( B ) ≥ ξ k +1 + δ 2 .
Revised: Case 1: Let B ∈ B (Γ) such that f ( B ) ≥ ξ k +1 + δ 2 .

Original: Then for all g ∈ N ( B ) holds f k ( g )
Revised: Then for all g ∈ N ( B ) holds f k ( g )

Original: = δ 2 .
Revised: = δ 2 .

Original: By construction exists a c (cid:48) ∈
Revised: By construction exists a c (cid:48) ∈

Original: ∆ k such that B ⊆ conv ( c (cid:48) ) .
Revised: ∆ k such that B ⊆ conv ( c (cid:48) ) .

Original: Using Lemma 4.3 we get
Revised: Using Lemma 4.3 we get

Original: [Equation]
Revised: [Equation]

Original: where [ p 1 , p 2 ] ∈ B ( R ≥ 0 ) .
Revised: where [ p 1 , p 2 ] ∈ B ( R ≥ 0 ) .

Original: Indeed, by case distinction:
Revised: Indeed, by case distinction:

Original: Case 2: Let B ∈ B (Γ) such that f ( B ) ≤ ξ k − δ 2 .
Revised: Case 2: Let B ∈ B (Γ) such that f ( B ) ≤ ξ k − δ 2 .

Original: Then for all g ∈ N ( B ) holds f k ( g ) = 0 .
Revised: Then for all g ∈ N ( B ) holds f k ( g ) = 0 .

Original: Further, B ∩ conv ( N ( c )) = ∅ for all c ∈ ∆ k because G is ﬁne enough.
Revised: Further, B ∩ conv ( N ( c )) = ∅ for all c ∈ ∆ k because G is ﬁne enough.

Original: Using Lemma 4.3 we obtain
Revised: Using Lemma 4.3 we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By construction we have n (cid:93)k ( B ) ⊆
Revised: By construction we have n (cid:93)k ( B ) ⊆