Original: T ROP E X : A N A LGORITHM FOR E XTRACTING L INEAR T ERMS IN D EEP N EURAL N ETWORKS
Revised: T ROP E X : A N A LGORITHM FOR E XTRACTING L INEAR T ERMS IN D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks with rectiﬁed linear (ReLU) activations are piecewise linear functions, partitioning the input space into an astronomically high number of linear regions.
Revised: Deep neural networks with rectiﬁed linear (ReLU) activations are piecewise linear functions, where hyperplanes partition the input space into an astronomically high number of linear regions.

Original: Attempts have been made to obtain insight into the nature of networks by countingand probing linear regions. However, due to their high computational complexity, previous algorithms have only been applied to study the linear regions ofsmall networks, almost exclusively using the simple MNIST data set for training, thereby limiting the conclusiveness of the observations.
Revised: Previous work focused on counting linear regions to measure the network’s expressive power and on analyzing geometric properties of the hyperplane conﬁgurations. In contrast, we aim to understand the impact of the linear terms on network performance, by examining the information encoded in their coefﬁcients.

Original: We propose TropEx, a  tropical algebra-inspired algorithm for extracting linear terms based on data
Revised: To this end, we derive TropEx, a nontrivial tropical algebrainspired algorithm to systematically extract linear terms based on data.

Original: , enabling an analysis of their coefﬁcients and allowing for a wide range of new experiments.

Original: In particular, we examine qualitative differences in  the features convolutional neural networks CNNs) and fully -connected networks ( FCNs extract from images
Revised: Applied to convolutional and fully-connected networks, our algorithm uncovers signiﬁcant differences in how the different networks utilize linear regions for generalization.

Original: CNNs seem to base their decisions on object shapes, whereas FCNs seem to heavily rely on the general colour distribution of an image. Moreover, comparing ﬁndings across different data sets lets us conclude that in contrast to previous practice , results on linear regions from MNIST may not generalise to more complex data sets.
Revised: This underlines the importance of systematic linear term exploration, to better understand generalization in neural networks trained with complex data sets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many of the most widely used neural network architectures, including VGG (Simonyan & Zisserman, 2014), GoogLeNet (Szegedy et al., 2015) and ResNet (He et al., 2016), make use of rectiﬁed linear activations (ReLU, (Hahnloser et al., 2000; Glorot et al., 2011), i.e., σ ( x ) = max { x, 0 } )) and are therefore piecewise linear functions.
Revised: Many of the most widely used neural network architectures, including VGG (Simonyan & Zisserman, 2014), GoogLeNet (Szegedy et al., 2015) and ResNet (He et al., 2016), make use of rectiﬁed linear activations (ReLU, (Hahnloser et al., 2000; Glorot et al., 2011), i.e., σ ( x ) = max { x, 0 } )) and are therefore piecewise linear functions.

Original: Despite  these functions being apparently simple, there is a lack of theoretical understanding of the factors that contribute to the success of these architectures.
Revised: Despite the apparent simplicity of these functions, there is a lack of theoretical understanding of the factors that contribute to the success of such architectures.

Original: Previous attempts of understanding piecewise linear network functions have focused on estimating the number of linear terms , which are the linear pieces that constitute the network function.
Revised: Previous attempts of understanding piecewise linear network functions have focused on estimating the number of linear terms , which are the linear pieces that constitute the network function.

Original: A linear region is being deﬁned as a maximally connected subset of the domain on which the network function is linear.
Revised: A linear region is being deﬁned as a maximally connected subset of the input space on which the network function is linear.

Original: Since computing the exact number of linear regions is intractable, work has focused on obtaining upper and lower bounds for this number(Arora et al., 2016; Serra et al., 2018;
Revised: Since computing the exact number of linear regions is intractable, work has focused on obtaining upper and lower bounds for this number (Arora et al., 2016; Serra et al., 2018;

Original: Pascanu et al., 2013; Raghu et al., 2017; Montufar et al., 2014; Montúfar, 2017; Zhang et al., 2018).
Revised: Pascanu et al., 2013; Raghu et al., 2017; Montufar et al., 2014; Montúfar, 2017; Zhang et al., 2018).

Original: To our knowledge, the currently best upper and lower bounds were calculated by Serra et al.and
Revised: To our knowledge, the currently best upper and lower bounds were calculated by Serra et al.

Original: Raghu et al.
Revised: Raghu et al.

Original: (2017) show these bounds are asymptotically tight.
Revised: (2017) show these bounds to be asymptotically tight.

Original: All  mentioned papers share the intuition that the number of linear regions of neural networks measure their expressivity.
Revised: All of the mentioned papers share the intuition that the number of linear regions of neural networks measures their expressivity.

Original: Since the bounds grow linearly in width and exponentially in depth, deep networks are interpreted to have greater representational power.
Revised: Since the bounds grow linearly in width and exponentially in depth, deep networks are interpreted to have greater representational power.

Original: However, these bounds are staggeringly high: The upper bound on the number of linear regions in (Serra et al., 2018) exceeds 10 300 even for the smallest networks we experimented on.
Revised: However, these bounds are staggeringly high: the upper bound on the number of linear regions in (Serra et al., 2018) exceeds 10 300 even for the smallest networks we experimented on.

Original: (There are approximately 10 80 atoms in the universe.)
Revised: (There are approximately 10 80 atoms in the universe.)

Original: For slightly larger networks, the upper bound exceeds 10 17000 whereas the lower bound exceeds 10linear regions.
Revised: For slightly larger networks, the upper bound exceeds 10 17000 whereas the lower bound exceeds 10 83 linear regions.

Original: This raises the question how many linear regions are actually necessary to conserve the performance attained on both training and unseen test data. For instance, it might of course be that the network divides its domain into linear regions that do not overlap with the data distribution. Further, the number of training samples is generally much smaller than the estimated number of linear regions, which means that lots of  linear regions contain no evidence of  data.
Revised: The number of training samples is generally much smaller than the estimated number of linear regions ( ≤ 10 6 ), so that almost none of the linear regions contains training data. This raises the question of how representative the number of linear regions is for network performance and how information extracted from training samples passes on to the many linear regions free of data for successful generalization to test data.

Original: There are indications that a high number of linear regions is not required for good network performance.
Revised: There are indications that a high number of linear regions is not required for good network performance.

Original: Frankle & Carbin (2019) point out that smaller networks perform similarly well as large neural networks, when a suitable initialization of the smaller network can be found from training the larger one.
Revised: Frankle & Carbin (2019) point out that smaller networks perform similarly well as large neural networks, when a suitable initialization of the smaller network can be found from training the larger one.

Original: Hence, the expressivity of the large network is helpful to explore the parameter space, but the small, less expressive network is sufﬁcient to achieve high accuracy.
Revised: Hence, the expressivity of the large network is helpful to explore the parameter space, but the small, less expressive network is sufﬁcient to achieve high accuracy.

Original: Lee et al.
Revised: Lee et al.

Original: and Croce et al.
Revised: and Croce et al.

Original: (2018) modify the training loss to encouragenetwork functions to have larger linear regions with the goal of robustness to adversarial attacks.
Revised: (2018) modify the training loss to encourage larger linear regions with the goal of robustness to adversarial attacks.

Original: Hanin & Rolnick (2019b;a) argue that in practice there are less linear regions than expected from the bounds and empirically investigate this for2D-cross sections for the MNIST dataset.
Revised: Hanin & Rolnick (2019b;c) argue that in practice there are less linear regions than expected from the bounds and empirically investigate this for the MNIST data set.

Original: All these observations question the explanatory power of astronomically high bounds for the number of linear regions.
Revised: All these observations question the explanatory power of astronomically high bounds for the number of linear regions.

Revised: More recently, the focus of research on linear regions has been shifting away from pure counting towards an understanding of the linear regions themselves.

Revised: Zhang & Wu (2020) study geometric properties of linear regions and notice that batch normalization and dropout, albeit leading to similar network accuracies, produce differently looking linear regions.

Revised: Our approach to the understanding of linear regions differs in that it investigates the linear coefﬁcients of linear regions.

Original: We propose TropEx , a tropical algebra-based algorithm extracting linear terms of the network function N (Figure 1) using a data set X .
Revised: To this end, we propose TropEx , a tropical algebra-based algorithm extracting linear terms of the network function N (Figure 1) using a data set X .

Original: TropEx outputs an extractednetwork function N ( X ) containing only the linear terms corresponding to regions on which data lies.
Revised: TropEx outputs an extracted function N ( X ) containing only the linear terms corresponding to regions on which data lies.

Original: As a result, N and N ( X ) agree on neighbourhoods of all data points.
Revised: As a result, N and N ( X ) agree on neighbourhoods of all data points.

Original: Our goal is to create a starting point for the study of generalization from a new viewpoint, i.e., the perspective of linear regions.
Revised: This creates a tool for the study of generalization from a new viewpoint, i.e., the perspective of linear regions and their coefﬁcients.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • A  computational framework representing tropical functions (Deﬁnition B.4) as matrices.
Revised: • A new computational framework representing tropical functions (Deﬁnition B.4) as matrices to efﬁciently perform tropical calculations appearing in networks with rectiﬁed linear activations.

Original: •  TropEx, an algorithm to  extract linear terms from piecewise linear network functions. • An open source implementation of the algorithm. A link to the code will be added upon publication.
Revised: • This framework allows us to derive TropEx, an algorithm to systematically extract linear terms from piecewise linear network functions (open source implementation provided upon publication).

Original: • Application of TropEx to fully connected (FCN) and convolutional networks (CNN) reveals that (i) consistently all training and test samples fall into different linear regions ; (ii) FCNs and CNNs differ in how they use linear regions free of training data, where CNNs, in
Revised: • An application of TropEx to fully connected (FCN) and convolutional networks (CNN) reveals that (i) consistently all training and test samples fall into different linear regions; (ii) Simple tasks (MNIST) can be solved with the few

Original:  contrast to FCNs, tend to learn  linear functions that are both diverse and meaningful on unseen data.
Revised: linear regions of training samples alone, while this does not hold for more complex data sets. (iii) FCNs and CNNs differ in how they use linear regions free of training data for their performance on test data: Several measures illustrate that CNNs, in contrast to FCNs, tend to learn more diverse linear terms. (iv) We conﬁrm that the number of linear regions is not a good indicator for network performance and show that the coefﬁcients of linear regions contain information on architecture and classiﬁcation performance.

Original:  B ACKGROUND AND O VERVIEW
Revised:  B ACKGROUND AND O VERVIEW

Original: It was recently shown by Charisopoulos & Maragos (2018); Zhang et al.
Revised: It was recently shown by Charisopoulos & Maragos (2018); Zhang et al.

Original: (2018) that ReLU neural network functions are the same as tropical rational maps .
Revised: (2018) that ReLU neural network functions are the same as tropical rational maps .

Original: Tropical rational maps are exactly those functions where each entry in the output vector can be written as a difference of maxima
Revised: Tropical rational maps are exactly those functions where each entry in the output vector can be written as a difference of maxima

Original: [Equation]
Revised: [Equation]

Original: where each a + ij , a − ij : R d → R is an afﬁne function with only positive coefﬁcients, taking the form x (cid:55)→ (cid:80) k w k x k + w 0 with all w i ∈ R ≥ 0 .
Revised: where each a + ij , a − ij : R d → R is an afﬁne function with only positive coefﬁcients, taking the form x (cid:55)→ (cid:80) k w k x k + w 0 with all w i ∈ R ≥ 0 .

Original: Since the number of terms in (1) dwarfs the number of atoms in the universe, it is impossible to obtain this expression in practice.
Revised: Since the number of terms in (1) dwarfs the number of atoms in the universe, it is impossible to obtain this expression in practice.

Original: Therefore, we only extract those terms that correspond to linear regions of data points.
Revised: Therefore, we only extract those terms that correspond to linear regions of data points.

Original: For a ﬁxed data point x ∈ X , the maximum of the network outputs can be written as max i N i ( x )
Revised: For a ﬁxed data point x ∈ X , the maximum of the network outputs can be written as max i N i ( x )

Original: = a + x ( x ) − a − x ( x ) , where a + x , a − x are the afﬁne functions such that a + x ( x ) ≥ a + ij ( x ) , a − x ( x ) ≥ a − ij ( x ) for all i, j . TropEx extracts a + x and a − x .
Revised: = a + x ( x ) − a − x ( x ) , where a + x , a − x are the afﬁne functions such that a + x ( x ) ≥ a + ij ( x ) , a − x ( x ) ≥ a − ij ( x ) for all i, j . TropEx extracts a + x and a − x .

Original: The extracted terms can be used to construct areduced tropical map N ( X ) , given by
Revised: The extracted terms can be used to construct a tropical map N ( X ) with maximally enlarged linear regions, given by

Original: [Equation]
Revised: [Equation]

Original: where there are D i data points x i 1 , . . .
Revised: where there are D i data points x i 1 , . . .

Original: , x i Di given label i by the original network.
Revised: , x i Di given label i by the original network.

Original: Being a tropical rational map, the function N ( X ) is again a ReLU neural network function by Zhang et al.
Revised: Being a tropical rational map, the function N ( X ) is again a ReLU neural network function by Zhang et al.

Original: The maximal component of the  extracted network N ( X ) agrees with the maximal component of the original network N (and hence the assigned label) in the neighbourhood of any data point x ∈ X .
Revised: The maximal entries of the two output vectors (hence also the assigned labels) of extracted function N ( X ) and original network N agree in the neighbourhood of any data point x ∈ X .

Original:  M ETHOD 3. M ATRIX R EPRESENTATION OF T ROPICAL R ATIONAL M APS
Revised:  M ETHOD 3. M ATRIX R EPRESENTATION OF T ROPICAL R ATIONAL M APS

Original: If one were to represent tropical rational maps symbolically on a computer, computations would be too slow.
Revised: If one were to represent tropical rational maps symbolically on a computer, computations would be too slow.

Original: Therefore, we present tropical rational maps as multi-dimensional arrays.
Revised: Therefore, we present tropical rational maps as multi-dimensional arrays.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: Given an afﬁne function a : R d 0 → R ; x (cid:55)→ (cid:80) k w k x k + w 0 , we will call the vector ( w 0 , w 1 , . . .
Revised: Given an afﬁne function a : R d 0 → R ; x (cid:55)→ (cid:80) k w k x k + w 0 , we will call the vector ( w 0 , w 1 , . . .

Original: , w d 0 ) its coefﬁcient vector , the scalar w 

Revised: , w d 0 ) its coefﬁcient vector , the scalar w

Original: its constant part and the vector ( w 1 , . .
Revised: its constant part and the vector ( w 1 , . .

Original: . , w d 0 ) its variable part .
Revised: . , w d 0 ) its variable part .

Original: We can represent functions
Revised: We can represent functions

Original: N i : R d 0 → R as in equation (1) in the following way: Let the rows of the matrix
Revised: N i : R d 0 → R as in equation (1) in the following way: Let the rows of the matrix

Original: A + i ∈ R
Revised: A + i ∈ R

Original: n × d 0 and the vector a + i ∈ R n × 1 be the variable and the constant parts of the afﬁne functions a + ij , respectively.
Revised: n × d 0 and the vector a + i ∈ R n × 1 be the variable and the constant parts of the afﬁne functions a + ij , respectively.

Original: (Analogously for A − i and a − i .)
Revised: (Analogously for A − i and a − i .)

Original: We can then deﬁne ( A
Revised: We can then deﬁne ( A

Original: + i , a + i )( x ) = max { A
Revised: + i , a + i )( x ) = max { A

Original: + i x + a + i } , where the maximum is taken over the rows of the resulting column vector.
Revised: + i x + a + i } , where the maximum is taken over the rows of the resulting column vector.

Original: If we deﬁne the formal quotient 1 of matrix-vector pairs by ( A + i , a + i ) /
Revised: If we deﬁne the formal quotient 1 of matrix-vector pairs by ( A + i , a + i ) /

Original: ( A − i , a − i )( x ) = max { A
Revised: ( A − i , a − i )( x ) = max { A

Original: + i x + a + i } − max { A − i x
Revised: + i x + a + i } − max { A − i x

Original: + a − i } , then N i ( x ) =
Revised: + a − i } , then N i ( x ) =

Original: ( A + i , a + i ) /
Revised: ( A + i , a + i ) /

Original: ( A − i , a − i )( x ) , giving us a matrixrepresentation of the function N i .
Revised: ( A − i , a − i )( x ) , giving us a matrixrepresentation of the function N i .

Original: The entire network function  can then be represented by a list (cid:0)(
Revised: An entire network function with s output dimensions can then be represented by a list (cid:0)

Original: [Equation] + [Equation]
Revised: ( A + i , a + i ) /

Original:  ≤ [Equation] ≤ [Equation]
Revised: ( A − i , a − i ) (cid:1) 1 ≤ i ≤ s .

Original: The advantage of the proposed matrix representation of tropical rational maps are natural operations performing calculations that arise for (concatenations of) layers of neural networks (see supplements).
Revised: The advantage of the proposed matrix representation of tropical rational maps are natural operations performing calculations that arise for (concatenations of) layers of neural networks (see supplements).

Original: A dense layer (cid:96) : R d 1 → R d 2 without ReLU activation is represented as a list (cid:0) (
Revised: A dense layer (cid:96) : R d 1 → R d 2 without ReLU activation is represented as a list (cid:0) (

Original: A + i , a + i ) /
Revised: A + i , a + i ) /

Original: ( A − i , a − i ) (cid:1) 1 ≤ i ≤ d 2 .
Revised: ( A − i , a − i ) (cid:1) 1 ≤ i ≤ d 2 .

Original: Denoting by W pos and W neg the positive and negative part of a matrix W , respectively, i.e. w pos ij = max { w ij , 0 } and w neg ij = max {− w ij , 0 } , the matrix representation of a single neuron n i ( x ) = max { w · x + b, 0 } is given by
Revised: Denoting by W pos and W neg the positive and negative part of a matrix W , respectively, i.e. w pos ij = max { w ij , 0 } and w neg ij = max {− w ij , 0 } , the matrix representation of a single neuron n i ( x ) = max { w · x + b, 0 } is given by

Original: [Equation]
Revised: [Equation]

Original: 3. E XTRACTING L INEAR T ERMS OF A C LASSIFICATION N ETWORK
Revised: 3. E XTRACTING L INEAR T ERMS OF A C LASSIFICATION N ETWORK

Original: We now consider a classiﬁcation neural network N with s labels.
Revised: We now consider a classiﬁcation neural network N with s labels.

Original: Wenow show that we can represent a network N constant matrix-pair ( A − , a − ) in the denominator.
Revised: We show that we can represent the network N with a matrix-vector pair ( A − , a − ) in the denominator that is constant over all output dimensions.

Revised: The proof is given in Section C.2 of the supplementary material.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: Let N : R d → R s be the function of a ReLU neural network for classiﬁcation with s output neurons.
Revised: Let N : R d → R s be the function of a ReLU neural network for classiﬁcation with s output neurons.

Original: Then there are afﬁne functions a + ij , a − j such that
Revised: Then there are afﬁne functions a + ij , a − j such that

Original: [Equation]
Revised: [Equation]

Original: where the maxima of the a − i ( x ) on the right is subtracted from each entry of the vector on the left.
Revised: where the maxima of the a − i ( x ) on the right is subtracted from each entry of the vector on the left.

Original: In terms of our matrices, N  can be represented by a list ( ( A + i , a + i ) /
Revised: In terms of our matrices the classiﬁcation network N with s labels can be represented by a list ( ( A + i , a + i ) /

Original: ( A − , a − ) )
Revised: ( A − , a − ) )

Original: 1 ≤ i ≤ s of matrix-pairs.
Revised: 1 ≤ i ≤ s of matrix-vector pairs.

Original: Theproof is given in Section C.2 of the supplementary material. How to perform classiﬁcation with an expression as in (3) . The labelthat a network assigns a data point is the argmax of the entries of N (  x ) .Subtracting the expression on the right in (3) from each
Revised: The label is then given by argmax 1 ≤ i ≤ s ( A + i , a + i )( x ) .

Original: [Table]
Revised: [Table]

Original: Data set X
Revised: set X

Original: = { ( x i k , i ) } 1 ≤ i ≤ s , with D i points of label
Revised: = { ( x i k , i ) } [Table] D i points [Table]

Original: i Output
Revised: label

Original: [Table]
Revised: [Table]

Original:  ExtractedNetwork Function N ( X ) =
Revised: Output : Extracted Function N ( X ) =

Original: ( ( A + i , a + i ) /
Revised: ( ( A + i , a + i ) /

Original: ( A − , a − ) )
Revised: ( A − , a − ) )

Original: 1 ≤ i ≤ s 1: W , b ← weight matrix and bias vector of last layer (cid:96)2: C − , c − ← column sums of W neg , b neg 3: C + ← W + C − , c + ← b + c − 4: for i = 1 to s do 5:
Revised: 1 ≤ i ≤ s 1: W , b ← weight matrix and bias vector of last layer (cid:96)

Original: A + i ← rep ( C + i • , D i ) , a + i ← rep ( c + i , D i ) (cid:46) Repetition D i times along the rows.
Revised: [Table]

Original: 6: A − ← rep ( C − ,D ) , a − ← rep ( c −  , D )
Revised: C − , c − ← column sums of W neg , b neg [Table] [Table] i =

Original: [Table]
Revised: [Table]

Original: = total
Revised: to s do

Original: [Table]
Revised: [Table]

Original: of data points
Revised: ← rep

Original: [Table]
Revised: [Table]

Original: A max ← maxima of the columns of all A + iand A − stacked 8: for last layer (cid:96) in N not yet used do
Revised: C + i

Original: [Table]
Revised: [Table]

Original: ( N ( X ) ,
Revised: D i ) ,

Original: [Table]
Revised: [Table]

Original: max ) ← 
Revised: ← rep

Original: [Table]
Revised: [Table]

Original: ( N ( X ),
Revised: c + i [Table] D i )

Original: [Table]
Revised: [Table]

Original: max ) according
Revised: i

Original: [Table]
Revised: [Table]

Original: max ← K + A max W pos a ← a + Ab a ← a + Ab
Revised: rows. [Table] [Table] a ← a + Ab a ← a + Ab

Original: A ← K + ConvTrans ( A , F )
Revised: A ← K + ConvTrans ( A , F )

Original: A ← K + AW
Revised: A ← K + AW

Original: Flatten
Revised: Flatten

Original: A , A max ← reshape to input shape of (cid:96)
Revised: A , A max ← reshape to input shape of (cid:96)

Original: L-ReLU α ← Leaky ReLU parameter
Revised: L-ReLU α ← Leaky ReLU parameter

Original: ReLU
Revised: ReLU

Original: A kj ← 0 if (cid:96) ( x k ) j
Revised: A kj ← 0 if (cid:96) ( x k ) j

Original: =
Revised: =

Original: A kj ← α · A kj if (cid:96) ( x k ) j ≤entry, leaves the argmax unchanged, so we can discard this expression for classiﬁcation and only use the vector on the left. In terms of our matrices, the label is given by [Equation]
Revised: A kj ← α · A kj if (cid:96) ( x k ) j ≤

Original: where the maximum is taken over the rows of the resulting vector. How to get the extracted function N ( X )  from thefull network N . 
Revised: How to get the extracted function N ( X ) of (2) from the network N . TropEx extracts, for each data point x k of label

Original: For each data point x k of label i ,TropEx extracts the afﬁne functions a + ij and a − l from the network representation in (3) such that a + ij ( x k ) ≥ a +˜ i ˜
Revised: i , the afﬁne functions a + ij and a − l from the network representation in (3) such that a + ij ( x k ) ≥ a +˜ i ˜

Original: j ( x k ) , a − l ( x k ) ≥ a ˜ l ( x k ) for all ˜ i, ˜ j, ˜ l .
Revised: j ( x k ) , a − l ( x k ) ≥ a ˜ l ( x k ) for all ˜ i, ˜ j, ˜ l .

Original: We start with the last layer of the network and inductively merge new layers into the existing matrix pairs.
Revised: We start with the last layer of the network and inductively merge new layers into the existing matrix pairs.

Original: The merge operations depend on the type of the layer and are shown in Table 1.
Revised: The merge operations depend on the type of the layer as shown in Table 1.

Original: Putting things together gives Algorithm 3.1, which has the extracted function N ( X ) as its output.
Revised: Putting things together gives Algorithm 3.1, which has the extracted function N ( X ) as its output.

Original: The run-time and storage complexitiesper data point correspond to 3 forward passes through the network.
Revised: The run-time and storage complexities point correspond to 3 forward passes through the network.

Original: Theorem 3.3 states that TropEx indeed results in a selection of linear terms based on a dataset of points and that the extracted tropical function agrees with the network on neighbourhoods of all training samples.
Revised: Theorem 3.3 states that TropEx indeed results in a selection of linear terms based on a data set of points and that the extracted tropical function agrees with the network on neighbourhoods of all these points.

Original: Its proof and the complete derivation of the algorithm are in the appendix, where we develop a framework that allows tropical calculations on  matrices that correspond to manipulations of tropical functions they represent.
Revised: Its proof and the complete, nontrivial derivation of the algorithm are in the appendix, where we develop a framework that enables calculations on tropical matrices that correspond to manipulations of tropical functions they represent.

Original: For illustrative purposes, we also present a worked-out example of applying TropEx to a toy neural network there.
Revised: For illustrative purposes, we also present a worked-out example of applying TropEx to a toy neural network there.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Let N =
Revised: Let N =

Original: ( N 1 , . .
Revised: ( N 1 , . .

Original: . , N s ) :
Revised: . , N s ) :

Original: R d → R s be the function of a ReLU neural network for classiﬁcation into s classes.
Revised: R d → R s be the function of a ReLU neural network for classiﬁcation into s classes.

Original: Let N ( X ) be the network obtained from Algorithm 3.1, applied to N using a data set X
Revised: Let N ( X ) be the network obtained from Algorithm 3.1, applied to N using a data set X

Original: = { ( x i k , i ) } 1 ≤ i ≤ s of D i data points x i k given label i by N .
Revised: = { ( x i k , i ) | 1 ≤ i ≤ s, 1 ≤ i k ≤ D i } of D i data points x i k given label i by N .

Original: (There are D points in total).
Revised: (There are D points in total).

Original: Then, (1) for all labels i , N ( X ) i
Revised: Then, (1) for all labels i , N ( X ) i

Original: [Equation]
Revised: [Equation]

Revised: max { a

Revised: [Equation]

Original:  where the a + ij and a − j are extracted from a representation of N i as in equation (3) ; and (2) every data point x k has a neighbourhood U k on which the maximum of the extracted function agrees with the maximal network output:
Revised: ( x ) , . . . , a − D ( x ) } , where the a + ij and a − j are extracted from a representation of N i as in Equation 3; and (2) every data point x k has a neighbourhood U k on which the maximum of the extracted function agrees with the maximal network output:

Original: [Equation]
Revised: [Equation]

Original: In particular,both N ( X ) and N classify all points in U k with the same label.
Revised: In particular, N ( X ) and N classify all points in U k by assigning the same label.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Revised: TropEx extracts a function containing only linear terms corresponding to regions on which the given data lies.

Revised: The extracted function agrees with the network on this data.

Revised: This allows us to compare linear regions of train and test data, to separate the network structure from the information contained in the linear coefﬁcients, and to test how well the linear coefﬁcients generalize to test data.

Original: Setup We train neural networks on MNIST (LeCun et al., 2010) and CIFAR10 (Krizhevsky, 2009).
Revised: Setup We train neural networks on MNIST (LeCun et al., 2010) and CIFAR10 (Krizhevsky, 2009).

Original: After training, we use training data points x ( tr ) to extract linear terms a + x ( tr ) and a − x ( tr ) and construct an extracted network N ( X ) as in equation (2).
Revised: After training, we use training data points x ( tr ) to extract linear terms a + x ( tr ) and a − x ( tr ) and construct an extracted function N ( X ) as in equation (2).

Original: For some experiments, we also extract linear terms a + x ( te ) and a − x ( te ) corresponding to test data points x ( te ) .
Revised: For some experiments, we also extract linear terms a + x ( te ) and a − x ( te ) corresponding to test data points x ( te ) .

Original: Regarding the architecture, we use fullyconnected networks, AllCNN-C from Springenberg et al.
Revised: Regarding the architecture, we use fullyconnected networks, AllCNN-C from Springenberg et al.

Original: and variations of VGG-B from
Revised: and variations of VGG-B from

Original: Simonyan & Zisserman (2014).
Revised: Simonyan & Zisserman (2014).

Original: Section D in the appendix summarizes all architectures we used in our experiments.
Revised: Section D in the appendix summarizes all architectures we used in our experiments.

Revised: If not stated otherwise, we use architectures Conv for CNNs and FCN8 for FCNs in our experiments.

Original: It is not our goal to train networks to state of the art-performance, but rather to compare variations of simple networks which are composed of the layers shown in Table 1.
Revised: It is not our goal to train networks to state of the art-performance, but rather to compare variations of simple networks which are composed of the layers shown in Table 1.

Original: All layers have ReLU activations except for the last layer where we apply a softmax output function into the ten respective classes.
Revised: All layers have ReLU activations except for the last layer where we apply a softmax output function into the ten respective classes.

Original: 3 We train ﬁve networks of each architecture to ensure the consistency of our results.
Revised: 3 We train ﬁve networks of each architecture to ensure the consistency of our results.

Original: Further details on the training setup can be found in the appendix.
Revised: Further details on the training setup can be found in the appendix.

Original: Counting of Linear Regions At ﬁrst, we investigate how training and test samples are distributed over the linear regions of the neural networks.
Revised: Counting of Linear Regions At ﬁrst, we investigate how training and test samples are distributed over the linear regions of the neural networks.

Original: For each data point x , let a x = a + x − a − x be network function corresponding to the linear region on which x lies.
Revised: For each data point x , let a x = a + x − a − x be network function corresponding to the linear region on which x lies.

Original: Since a x (cid:54) = a x (cid:48) for all training and test points x , x (cid:48) , we see that all points lie in different linear regions.
Revised: Since a x (cid:54) = a x (cid:48) for all training and test points x , x (cid:48) , we see that all points lie in different linear regions.

Original: This is not a result of overﬁtting during training: All data points also occupy different regions when we count the number of linear regions after 1, 3, 5, 10, 15, 20, 30, 40 and 50 epochs of training on CIFAR10, and from epochs 1 to 20 on MNIST.
Revised: This is not a result of overﬁtting during training: All data points also occupy different regions when we count the number of linear regions after 1, 3, 5, 10, 15, 20, 30, 40 and 50 epochs of training on CIFAR10, and from epochs 1 to 20 on MNIST.

Original: Therefore we conclude that generalization capabilities of neural networks cannot be explained by test samples falling into the same linear region as training samples (or, in other words, by test samples inducing  the same activation pattern as training samples).
Revised: Therefore we conclude that generalization capabilities of neural networks cannot be explained by test samples falling into the same linear region as training samples (or, in other words, by test samples inducing exactly the same activation pattern as training samples).

Revised: Examining Function Coefﬁcients With test samples falling into different regions than training samples, it is conceivable that neighboring regions could still be “nearly identical” and test samples would fall into such neighboring regions of training samples.

Revised: To clarify, we ran experiments to test similarity of linear regions for test samples before and after reduction.

Original: Accuracy of the Extracted Functions As predicted by Theorem 3.3, the maximum of  the extracted function N ( X ) agrees with the maximum of the original network on all training points for each of our networks.
Revised: We take a test sample and extract the linear coefﬁcient of its linear region in both the original network N and in the extracted function N ( X ) . For these linear coefﬁcient vectors, we calculate the (i) angle and (ii) Euclidean norm difference. Figure 2 shows both values for all test samples, where we differentiate between those test points x ( te ) that get correctly classiﬁed by N ( X ) (in blue) and those that get a wrong label (in red). We observe a clear difference between the CNN and the FCN.

Revised: For CNNs, the coefﬁcient vectors of both training and test afﬁne functions are all close to orthogonal for correctly as well as incorrectly classiﬁed points, For FCNs, angle and distance of correctly classiﬁed points are smaller than for incorrectly classiﬁed ones, but still far away from zero and therefore rule out the possibility of test samples falling into very similar neighboring regions.

Revised: Finally, instead of comparing linear coefﬁcients, we also tested the similarity in activation patterns before and after extraction.

Revised: The results in section E.4 of the appendix show that in each layer approx 80% of neuron activations agree between test and training region, showing that also the activation patterns of test samples deviate considerably and generalization cannot be explained by very similar activation patterns.

Revised: Accuracy of the Extracted Functions As predicted by Theorem 3.3, the maximum of the extracted function N ( X ) agrees with the maximum of the original network on all training points for each of our networks.

Original: In particular, network and extracted function assign the same label to all training points.
Revised: In particular, network and extracted function assign the same label to each training point.

Original: To investigate how well the coefﬁcients of these linear regions generalize to unseen data, we compare test accuracy of network and extracted function in Table 2.
Revised: To investigate how well the coefﬁcients of these linear regions generalize to unseen data, we compare test accuracy of network and extracted function in Table 2.

Original: We see a consistent difference between
Revised: We see a consistent difference between

Revised: [Table]

Original: CNNs and FCNs across both data sets and all architectures: There is a drastic drop in the test accuracy of the CNNs, as opposed to a relatively small drop in the accuracy of the FCNs.
Revised: CNNs and FCNs across both data sets and all architectures: There is a drastic drop in the test accuracy of the CNNs, as opposed to a relatively small drop in the accuracy of the FCNs.

Original: Interestingly, for
Revised: Interestingly, for

Original: MNIST, the extracted tropical function has almost the same test accuracy as the original network.
Revised: MNIST, the extracted tropical function has almost the same test accuracy as the original network.

Original: This is surprising as all known bounds on the number of linear regions of the original network suggest numbers of the order of 10 80 up to over 10 17000 from which we only observe 60.000 after reduction.
Revised: This is surprising as all known bounds on the number of linear regions of the original network suggest numbers of the order of 10 80 up to over 10 17000 from which we only observe 60.000 after reduction.

Original: Hence, for fully connected networks on a simple task, the coefﬁcients used on training data generalize well to test data, but for complex data, the learned coefﬁcients generalize worse.
Revised: Hence, for fully connected networks on a simple task, the coefﬁcients used on training data generalize well to test data [Table] but for complex data, the learned coefﬁcients generalize worse.

Revised: This is remarkable, since previous studies (Hanin & Rolnick, 2019b;a; Zhang & Wu, 2020) of linear regions were forced to base their experiments on small data sets (or small networks) for computational reasons

Revised: [Table]

Revised: and it seems that care must be taken when generalizing observations to more complex tasks.

Original: Moreover, the results reveal a difference between FCNs and CNNs that we further investigate.
Revised: Moreover, the results reveal another difference between FCNs and CNNs that we further investigate.

Original: Transformation Process Figure 2, left shows how test accuracy and agreement with the original network develop during the transformation from full network N to its extracted form N ( X ) .
Revised: Transformation Process Figure 3, left shows how test accuracy and agreement with the original network develop during the transformation from full network N to its extracted form N ( X ) .

Original: Starting with the last layer, TropEx iteratively merges layers into an extractednetwork function.
Revised: Starting with the last layer, TropEx iteratively merges layers into an extracted function.

Original: The x-axis shows which layers have been merged to an extracted tropical function at each step.
Revised: The x-axis shows which layers have been merged to an extracted tropical function at each step.

Original: The graph shows accuracy values for passing test samples through the original network until the layer representing the input to the extracted tropical function and then applying this extracted function.
Revised: The graph shows accuracy values for passing test samples through the original network until the layer that represents the input to the extracted tropical function and then applying this extracted function.

Original: At the right end of the plots all layers have been merged to the tropical function N ( X ) .
Revised: At the right end of the plots all layers have been merged to the tropical function N ( X ) .

Original: There is a clear difference between fully-connected networks and CNNs, which is consistent over all networks (Figure 2, right).[Table]
Revised: There is a clear difference between fully-connected networks and CNNs, which is consistent over all networks (Figure 3, right).

Original: architectures are described in the appendix.

Original: The fully connected network Wide and the convolutional network Narrow have exactly the same number of nodes after each parameter layer, but the drop in test accuracy for Narrow is almost 5 times the drop for Wide (36.6% vs 8.1%).
Revised: The fully connected network Wide and the convolutional network Narrow have exactly the same number of nodes after each parameter layer, but the drop in test accuracy for Narrow is almost 5 times the drop for Wide (36.6% vs 8.1%).

Original: Starting with a much superior performance, after reduction of linear terms the convolutional network is outperformed by the fully connected network.
Revised: Starting with a much superior performance, after reduction of linear terms the convolutional network is outperformed by the fully connected network.

Original: Since Narrow has only few connections between its nodes and Wide is fully-connected, it would be reasonable to assume that the number of linear regions of Wide is much greater than the number of linear regions of
Revised: Since Narrow has only few connections between its nodes and Wide is fully-connected, it would be reasonable to assume that the number of linear regions of Wide is much greater than the number of linear regions of

Original: Narrow .
Revised: Narrow , since its theoretical upper bound is easily seen to be higher.

Original: Hence, it would be expected that an extraction of a ﬁxed number of linear terms resulted in a smaller change of results for Narrow and the results werebe reversed.
Revised: Hence, it would be expected that an extraction of a ﬁxed number of linear terms resulted in a smaller change of results for Narrow and the results were reversed.

Original: This contradicts our intuition about how CNNs and FCNs work from a perspective of  linear regions.
Revised: This contradicts our intuition about how CNNs and FCNs work from the perspective of network expressivity in terms of the number of linear regions.

Original: Network Training We compare the performance of extracted functions N ( X ) during training of the network N .
Revised: Network Training We compare the performance of extracted functions N ( X ) during training of the network N .

Original: Figure 3 displays the test accuracy (mean and standard deviation over 5 networks) of the extracted function and the agreement of label assignments of extracted form with the original network function for Narrow (CNN) and Wide (fully connected) for several epochs.
Revised: Figure 4 displays the test accuracy (mean and standard deviation over 5 networks) of the extracted function and the agreement of label assignments of extracted form with the original network function for Narrow (CNN) and Wide (fully connected) for several epochs.

Original: The difference between fully connected and convolutional networks is here even more striking.
Revised: The difference between fully connected and convolutional networks is here even more striking.

Original: For the CNN, the agreement between extracted tropical function and original network function falls rapidly after only one epoch and only slightly reduces from there.
Revised: For the CNN, the agreement between extracted tropical function and original network function falls rapidly after only one epoch and only slightly reduces from there.

Original: For the fully connected network, the agreement decreases slowly over the entire 50 training epochs and it never reaches a value as low as the CNN after its ﬁrst epoch.
Revised: For the FCN, the agreement decreases slowly over the entire 50 training epochs and it never reaches a value as low as the CNN after its ﬁrst epoch.

Original:  The extractednetwork functions all share the number of linear terms, hence their difference in performance must be explained by the coefﬁcient values.
Revised: Information encoded in linear coefﬁcients The extracted functions all share the number of linear terms, hence their difference in performance must be explained by the coefﬁcient values.

Original: With this in mind, we attempt an interpretation of these results.
Revised: With this in mind, we attempt an interpretation of the above results and hypothesize that the differences lies in how FCN and CNN store important information for the classiﬁcation task in the linear coefﬁcients of linear regions.

Original: We hypothesize that the  structure on the weight spaceimposed by  weight sharing in the CNN results in the incapability to use dominant linear coefﬁcients for correct label assignments. Instead, the CNN uses the imposed structure on the weights for generalization (such as re-occurrence of ﬁlters at many positions).
Revised: An FCN has the full freedom to compose weights to tailored coefﬁcients of linear regions, whereas CNNs impose a structure on the weight space by ﬁlters and weight sharing, which results in the incapability to compose tailored linear coefﬁcients for correct label assignments. Instead, the structural properties of convolutions composing the network function play a signiﬁcant role for generalization.

Original: By extracting linear terms we remove some structure, which strongly changes the outcome on test data as the coefﬁcients in linear regions alone are limited in meaning.
Revised: By extracting linear terms we remove the structure, which strongly changes the outcome on test data as the coefﬁcients of linear regions alone are limited in meaning.

Original: In contrast, the FCN has full freedom to use linear coefﬁcients tailored to detect local speciﬁcs (such as green dot at a location speciﬁc to a certain class).

Original: This leads the label prediction to strongly rely on speciﬁc weight combinations.

Original: As training progresses, to achieve higher accuracy, the network learns to use structure instead of “overﬁtting" on easy samples. Hence, structural use slowly outweighs thereliance on coefﬁcient values and removal of this structure leads to a decrease in performance of the extracted function increasingly similar to the observations for CNNs.
Revised: As training progresses, to achieve higher accuracy, the FCN reduces the information stored in linear coefﬁcients and also learns to use some structure, so that the removal of this structure could explain the decrease in performance of the extracted function.

Original: An experiment, where we visually inspect misclassiﬁed images (Appendix E.2) is in line with this interpretation. Examining Function Coefﬁcients With test samples falling into different regions than training samples, it is conceivable that neighboring regions could still be “nearly identical” and test samples would fall into such neighboring regions of training samples. To clarify, we ran an experiment to test similarity of linear regions for test samples before and after reduction intwo different ways. We take a test sample and extract the linear coefﬁcient ofits linear region in both the original network N and in the extracted network N ( X ) .
Revised: An experiment, where we visually inspect misclassiﬁed images (Appendix E.1) is in line with this interpretation suggesting that the object shape is encoded in the linear coefﬁcients of the FCNs, but for CNNs only simple features such as background color are encoded in the linear coefﬁcients of linear regions.

Original: For these linear coefﬁcient vectors, we calculate the (i) angle and (ii) Euclidean norm difference. Figure 4 shows both values for all test samples, where we differentiate between those test points x ( te ) that get correctly classiﬁed by N ( X ) (in blue) and those thatget a wrong label (in red). For CNNs, the coefﬁcient vectors of the both training and test afﬁne functions are all close to orthogonal forcorrectly as well as incorrectly classiﬁed points, For FCNs, angle and distanceof correctly classiﬁed points are smaller than for incorrectly classiﬁed ones.
Revised: Re-training the network The observation of smaller angles for FCNs in Figure 2 further supports our interpretation that coefﬁcient values of linear regions play a larger role in classiﬁcation for FCNs, since smaller angles together with small Euclidean distance are explained by a similarity of the coefﬁcient values.

Original: These results rule out the possibility of test samples falling into very similar neighboring regions. It further supports the interpretation that generalization inCNNs is more dependent on thestructure imposed by, e.g. weight sharing, and less dependent on dominant linear coefﬁcient values.
Revised: This suggests to also compare the similarity of linear coefﬁcients after re-training the network in order to further understand the information encoded in the linear coefﬁcients.

Original: While the angles observed for the CNN are almost consistently orthogonal, there is a clear correlation for the FCN between small angles and correct classiﬁcation by the extracted function.
Revised: Again,we ﬁnd that the coefﬁcient vectors of two separately trained CNNs are close to orthogonal, whereas both angles and distances are considerably smaller for FCNs.

Original: D ISCUSSION AND C ONCLUSION
Revised: 5 Plots are shown in Appendix E.5.

Original: Potential for Future Work. • Adversarial Attacks. Since large linear regions add adversarial robustness (Croce et al., 2018) and TropEx outputs a network function with maximally enlarged linear regions, it may provide a tool for studying various adversarial attacks.
Revised: For each dimension we additionally compute the Pearson correlation between the linear coefﬁcients of two separately trained networks over all training samples.

Original: It would also be interesting to see whether adversarial attacks

Original: can harm the performance of the extracted function N ( X ) on training data. • Network Pruning. Our experiments show that removing the structure from a network N causes a signiﬁcant drop in test accuracy while keeping the training accuracy unchanged. Zhang et al.
Revised: We reduce the resulting vector to a single number by averaging the correlation factors over the dimensions.

Original: (2018) show that tropical rational maps are the same as ReLU networks, so it is possible to obtain a new architecture N (cid:48) from N ( X ) . From a perspective of linear regions, TropEx outputs a minimal function that is unchanged on all training data. Since different network architectures can describe the same network function, it remains to understand what additional structure needs to be imposed on the smaller architecture to obtain high test accuracy as well. This opens the possibility for a pruning technique with unchanged predictions on the data it was trained on.
Revised: We experiment with two pairs of FCNs and two pairs of CNNs trained on CIFAR10, and one pair of FCNs trained on MNIST.

Original: • Generalization theory. Deep neural networks can learn data perfectly (Zhang et al., 2016), which results in a network function thatcorrectly classiﬁes any training sample. For such over-parameterized models, Belkin et al. (2018) study the performance on test data from a viewpoint of interpolation.
Revised: Figure 5 shows the evolution of the correlation during training, conﬁrming that the similarity of coefﬁcient values is also much larger as measured by correlation.

Original: Our extracted function N ( X ) agrees with the network N on the data set X , hence their difference ingeneralization must depend on how they interpolate between the training data X By observing consistent differences between CNNs and FCNs, our results suggest that the architecture plays a large role in this interpolation process. Tailored experiments with TropEx can give more insight on the connection
Revised: Interestingly, for the FCN we notice a drastic jump in the correlation values to a larger value precisely when the training accuracy reaches 100% , showing a signiﬁcant correlation of linear coefﬁcients after re-training after convergence.

Original: between network architecture and interpolation performance and support the derivation of bounds on the generalization error for neural networks

Original:  Conclusion
Revised:  C ONCLUSION

Original: The function of a ReLU network is piecewise linear, with an astronomically high number of linear regions.
Revised: The function of a ReLU network is piecewise linear, with an astronomically high number of linear regions.

Original: We introduced TropEx, an algorithm for extracting linear regions based on data points.
Revised: We introduced TropEx, an algorithm to systematically extract linear regions based on data points.

Original: The derivationof the algorithm is based on a matrix representation of tropical functions.
Revised: The derivation is based on a matrix representation of tropical functions that supports efﬁcient algorithmic development.

Original: Applying
Revised: TropEx enables investigations of the linear components of piecewise linear network functions: By extracting the network’s linear terms, the algorithm allows us to compare training and test regions and to systematically analyze their linear coefﬁcients.

Original:  TropEx to fully-connected and convolutional architectures shows a drastic drop in test accuracy for CNNs, compared to a relatively small drop for FCNs.
Revised: Applying TropEx to fully-connected and convolutional architectures shows signiﬁcant differences between linear regions of CNNs and FCNs.

Revised: Other possible use cases are outlined in Appendix G.

Original: The reason for this difference can be found in the coefﬁcients of the extracted linear terms: For CNNs, theenlarged linearregions of training data on which test data lies after extraction differ in terms of angle and distance, whereas for FCNs they are more similar, in particular for those points where the label is unchanged after the extraction. These ﬁndings indicate that research on linear regions should shift its focus from counting to an understanding of their interplay.
Revised: Our ﬁndings indicate a potential beneﬁt from shifting focus from counting linear regions to an understanding of their interplay, as differences between CNNs and FCNs may be found in the coefﬁcients of the extracted linear terms. Several measures of similarity indicate that the linear terms of CNNs are more diverse than those of FCNs and suggest that CNNs efﬁciently exploit the structure imposed by their architecture, whereas FCNs rely on encoding information in the values of linear coefﬁcients.