Original: T ROP E X : A N A LGORITHM FOR E XTRACTING L INEAR T ERMS IN D EEP N EURAL N ETWORKS
Revised: T ROP E X : A N A LGORITHM FOR E XTRACTING L INEAR T ERMS IN D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks with rectiﬁed linear (ReLU) activations are piecewise linear functions, where hyperplanes partition the input space into an astronomically high number of linear regions.
Revised: Deep neural networks with rectiﬁed linear (ReLU) activations are piecewise linear functions, where hyperplanes partition the input space into an astronomically high number of linear regions.

Original: Previous work focused on counting linear regions to measure the network’s expressive power and on analyzing geometric properties of the hyperplane conﬁgurations.
Revised: Previous work focused on counting linear regions to measure the network’s expressive power and on analyzing geometric properties of the hyperplane conﬁgurations.

Original: In contrast, we aim to understand the impact of the linear terms on network performance, by examining the information encoded in their coefﬁcients.
Revised: In contrast, we aim to understand the impact of the linear terms on network performance, by examining the information encoded in their coefﬁcients.

Original: To this end, we derive TropEx, a nontrivial tropical algebrainspired algorithm to systematically extract linear terms based on data.
Revised: To this end, we derive TropEx, a nontrivial tropical algebrainspired algorithm to systematically extract linear terms based on data.

Original: Applied to convolutional and fully-connected networks, our algorithm uncovers signiﬁcant differences in how the different networks utilize linear regions for generalization.
Revised: Applied to convolutional and fully-connected networks, our algorithm uncovers signiﬁcant differences in how the different networks utilize linear regions for generalization.

Original: This underlines the importance of systematic linear term exploration, to better understand generalization in neural networks trained with complex data sets
Revised: This underlines the importance of systematic linear term exploration, to better understand generalization in neural networks trained with complex data sets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many of the most widely used neural network architectures, including VGG (Simonyan & Zisserman, 2014), GoogLeNet (Szegedy et al., 2015) and ResNet (He et al., 2016), make use of rectiﬁed linear activations (ReLU, (Hahnloser et al., 2000; Glorot et al., 2011), i.e., σ ( x ) = max { x, 0 } )) and are therefore piecewise linear functions.
Revised: Many of the most widely used neural network architectures, including VGG (Simonyan & Zisserman, 2014), GoogLeNet (Szegedy et al., 2015) and ResNet (He et al., 2016), make use of rectiﬁed linear activations (ReLU, (Hahnloser et al., 2000; Glorot et al., 2011), i.e., σ ( x ) = max { x, 0 } )) and are therefore piecewise linear functions.

Original: Despite the apparent simplicity of these functions, there is a lack of theoretical understanding of the factors that contribute to the success of such architectures.
Revised: Despite the apparent simplicity of these functions, there is a lack of theoretical understanding of the factors that contribute to the success of such architectures.

Original: Previous attempts of understanding piecewise linear network functions have focused on estimating the number of linear terms , which are the linear pieces  that constitute the network function.
Revised: Previous attempts of understanding piecewise linear network functions have focused on estimating the number of linear terms , which are the linear pieces (afﬁne functions) that constitute the network function.

Original: A linear region is being deﬁned as a maximally connected subset of the input space on which the network function is linear.
Revised: A linear region is being deﬁned as a maximally connected subset of the input space on which the network function is linear.

Original: Since computing the exact number of linear regions is intractable, work has focused on obtaining upper and lower bounds for this number (Arora et al., 2016; Serra et al., 2018;
Revised: Since computing the exact number of linear regions is intractable, work has focused on obtaining upper and lower bounds for this number (Arora et al., 2016; Serra et al., 2018;

Original: Pascanu et al., 2013; Raghu et al., 2017; Montufar et al., 2014; Montúfar, 2017; Zhang et al., 2018).
Revised: Pascanu et al., 2013; Raghu et al., 2017; Montufar et al., 2014; Montúfar, 2017; Zhang et al., 2018).

Original: To our knowledge, the currently best upper and lower bounds were calculated by Serra et al.
Revised: To our knowledge, the currently best upper and lower bounds were calculated by Serra et al.

Original: Raghu et al.
Revised: Raghu et al.

Original: (2017) show these bounds to be asymptotically tight.
Revised: (2017) show these bounds to be asymptotically tight.

Original: All of the mentioned papers share the intuition that the number of linear regions of neural networks measures their expressivity.
Revised: All of the mentioned papers share the intuition that the number of linear regions of neural networks measures their expressivity.

Original: Since the bounds grow linearly in width and exponentially in depth, deep networks are interpreted to have greater representational power.
Revised: Since the bounds grow linearly in width and exponentially in depth, deep networks are interpreted to have greater representational power.

Original: However, these bounds are staggeringly high: the upper bound on the number of linear regions in (Serra et al., 2018) exceeds 10 300 even for the smallest networks we experimented on.
Revised: However, these bounds are staggeringly high: the upper bound on the number of linear regions in (Serra et al., 2018) exceeds 10 300 even for the smallest networks we experimented on.

Original: (There are approximately 10 80 atoms in the universe.)
Revised: (There are approximately 10 80 atoms in the universe.)

Original: For slightly larger networks, the upper bound exceeds 10 17000 whereas the lower bound exceeds 10 83 linear regions.
Revised: For slightly larger networks, the upper bound exceeds 10 17000 whereas the lower bound exceeds 10 83 linear regions.

Original: The number of training samples is generally much smaller than the estimated number of linear regions ( ≤ 10 6 ), so that almost none of the linear regions contains training data.
Revised: The number of training samples is generally much smaller than the estimated number of linear regions ( ≤ 10 6 ), so that almost none of the linear regions contains training data.

Original: This raises the question of how representative the number of linear regions is for network performance and how information extracted from training samples passes on to the many linear regions free of data for successful generalization to test data.
Revised: This raises the question of how representative the number of linear regions is for network performance and how information extracted from training samples passes on to the many linear regions free of data for successful generalization to test data.

Original: There are indications that a high number of linear regions is not required for good network performance.
Revised: There are indications that a high number of linear regions is not required for good network performance.

Original: Frankle & Carbin (2019) point out that smaller networks perform similarly well as large neural networks, when a suitable initialization of the smaller network can be found from training the larger one.
Revised: Frankle & Carbin (2019) point out that smaller networks perform similarly well as large neural networks, when a suitable initialization of the smaller network can be found from training the larger one.

Original: Hence, the expressivity of the large network is helpful to explore the parameter space, but the small, less expressive network is sufﬁcient to achieve high accuracy.
Revised: Hence, the expressivity of the large network is helpful to explore the parameter space, but the small, less expressive network is sufﬁcient to achieve high accuracy.

Original: Lee et al.
Revised: Lee et al.

Original: and Croce et al.
Revised: and Croce et al.

Original: (2018) modify the training loss to encourage larger linear regions with the goal of robustness to adversarial attacks.
Revised: (2018) modify the training loss to encourage larger linear regions with the goal of robustness to adversarial attacks.

Original: Hanin & Rolnick (2019b;c) argue that in practice there are less linear regions than expected from the bounds and empirically investigate this for the MNIST data set.
Revised: Hanin & Rolnick (2019b;c) argue that in practice there are less linear regions than expected from the bounds and empirically investigate this for the MNIST data set.

Original: All these observations question the explanatory power of astronomically high bounds for the number of linear regions.
Revised: All these observations question the explanatory power of astronomically high bounds for the number of linear regions.

Original: More recently, the focus of research on linear regions has been shifting away from pure counting towards an understanding of the linear regions themselves.
Revised: More recently, the focus of research on linear regions has been shifting away from pure counting towards an understanding of the linear regions themselves.

Original: Zhang & Wu (2020) study geometric properties of linear regions and notice that batch normalization and dropout, albeit leading to similar network accuracies, produce differently looking linear regions.
Revised: Zhang & Wu (2020) study geometric properties of linear regions and notice that batch normalization and dropout, albeit leading to similar network accuracies, produce differently looking linear regions.

Original: Our approach to the understanding of linear regions differs in that it investigates the linear coefﬁcients of linear regions.
Revised: Our approach to the understanding of linear regions differs in that it investigates the linear coefﬁcients of linear regions.

Original: To this end, we propose TropEx , a tropical algebra-based algorithm extracting linear terms of the network function N (Figure 1) using a data set X .
Revised: To this end, we propose TropEx , a tropical algebra-based algorithm extracting linear terms of the network function N (Figure 1) using a data set X .

Original: TropEx outputs an extracted function N ( X ) containing only the linear terms corresponding to regions on which data lies.
Revised: TropEx outputs an extracted function N ( X ) containing only the linear terms corresponding to regions on which data lies.

Original: As a result, N and N ( X ) agree on neighbourhoods of all data points.
Revised: As a result, N and N ( X ) agree on neighbourhoods of all data points.

Original: This creates a tool for the study of generalization from a new viewpoint, i.e., the perspective of linear regions and their coefﬁcients.
Revised: This creates a tool for the study of generalization from a new viewpoint, i.e., the perspective of linear regions and their coefﬁcients.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • A new computational framework representing tropical functions (Deﬁnition B.4) as matrices to efﬁciently perform tropical calculations appearing in networks with rectiﬁed linear activations.
Revised: • A new computational framework representing tropical functions (Deﬁnition B.4) as matrices to efﬁciently perform tropical calculations appearing in networks with rectiﬁed linear activations.

Original: • This framework allows us to derive TropEx, an algorithm to systematically extract linear terms from piecewise linear network functions (open source implementation provided upon publication).
Revised: • This framework allows us to derive TropEx, an algorithm to systematically extract linear terms from piecewise linear network functions (open source implementation provided upon publication).

Original: • An application of TropEx to fully connected (FCN) and convolutional networks (CNN) reveals that (i) consistently all training and test samples fall into different linear regions; (ii) Simple tasks (MNIST) can be solved with the few
Revised: • An application of TropEx to fully connected (FCN) and convolutional networks (CNN) reveals that (i) consistently all training and test samples fall into different linear regions; (ii) Simple tasks (MNIST) can be solved with the few

Original: linear regions of training samples alone, while this does not hold for more complex data sets.
Revised: linear regions of training samples alone, while this does not hold for more complex data sets.

Original: (iii) FCNs and CNNs differ in how they use linear regions free of training data for their performance on test data: Several measures illustrate that CNNs, in contrast to
Revised: (iii) FCNs and CNNs differ in how they use linear regions free of training data for their performance on test data: Several measures illustrate that CNNs, in contrast to

Original: FCNs, tend to learn more diverse linear terms.
Revised: FCNs, tend to learn more diverse linear terms.

Original: (iv) We conﬁrm that the number of linear regions is not a good indicator for network performance and show that the coefﬁcients of linear regions contain information on architecture and classiﬁcation performance.
Revised: (iv) We conﬁrm that the number of linear regions is not a good indicator for network performance and show that the coefﬁcients of linear regions contain information on architecture and classiﬁcation performance.

Original:  B ACKGROUND AND O VERVIEW
Revised:  B ACKGROUND AND O VERVIEW

Original: It was recently shown by Charisopoulos & Maragos (2018); Zhang et al.
Revised: It was recently shown by Charisopoulos & Maragos (2018); Zhang et al.

Original: (2018) that ReLU neural network functions are the same as tropical rational maps .
Revised: (2018) that ReLU neural network functions are the same as tropical rational maps .

Original: Tropical rational maps are exactly those functions where each entry in the output vector can be written as a difference of maxima
Revised: Tropical rational maps are exactly those functions where each entry in the output vector can be written as a difference of maxima

Original: [Equation]
Revised: [Equation]

Original: where each a + ij , a − ij : R d → R is an afﬁne function with only positive coefﬁcients, taking the form x (cid:55)→ (cid:80) k w k x k + w 0 with all w i ∈ R ≥ 0 .
Revised: where each a + ij , a − ij : R d → R is an afﬁne function with only positive coefﬁcients, taking the form x (cid:55)→ (cid:80) k w k x k + w 0 with all w i ∈ R ≥ 0 .

Original: Since the number of terms in (1) dwarfs the number of atoms in the universe, it is impossible to obtain this expression in practice.
Revised: Since the number of terms in (1) dwarfs the number of atoms in the universe, it is impossible to obtain this expression in practice.

Original: Therefore, we only extract those terms that correspond to linear regions of data points.
Revised: Therefore, we only extract those terms that correspond to linear regions of data points.

Original: For a ﬁxed data point x ∈ X , the maximum of the network outputs can be written as max i N i ( x )
Revised: For a ﬁxed data point x ∈ X , the maximum of the network outputs can be written as max i N i ( x )

Original: = a + x ( x ) − a − x ( x ) , where a + x , a − x are the afﬁne functions such that a + x ( x ) ≥ a + ij ( x ) , a − x ( x ) ≥ a − ij ( x ) for all i, j . TropEx extracts a + x and a − x .
Revised: = a + x ( x ) − a − x ( x ) , where a + x , a − x are the afﬁne functions such that a + x ( x ) ≥ a + ij ( x ) , a − x ( x ) ≥ a − ij ( x ) for all i, j . TropEx extracts a + x and a − x .

Original: The extracted terms can be used to construct a tropical map N ( X ) with maximally enlarged linear regions, given by
Revised: The extracted terms can be used to construct a tropical map N ( X ) with maximally enlarged linear regions, given by

Original: [Equation]
Revised: [Equation]

Original: where there are D i data points x i 1 , .. .
Revised: where there are D i data points x k 1 , .

Original: , x i Di given label i by the original network.
Revised: , x k Di given label i by the original network.

Original: Being a tropical rational map, the function N ( X ) is again a ReLU neural network function by Zhang et al.
Revised: Being a tropical rational map, the function N ( X ) is again a ReLU neural network function by Zhang et al.

Original: The maximal entries of the two output vectors (hence also the assigned labels) of extracted function
Revised: The maximal entries of the two output vectors (hence also the assigned labels) of extracted function

Original: N ( X ) and original network N agree in the neighbourhood of any data point x ∈ X .
Revised: N ( X ) and original network N agree in the neighbourhood of any data point x ∈ X .

Original:  M ETHOD 3. M ATRIX R EPRESENTATION OF T ROPICAL R ATIONAL M APS
Revised:  M ETHOD 3. M ATRIX R EPRESENTATION OF T ROPICAL R ATIONAL M APS

Original: If one were to represent tropical rational maps symbolically on a computer, computations would be too slow.
Revised: If one were to represent tropical rational maps symbolically on a computer, computations would be too slow.

Original: Therefore, we present tropical rational maps as multi-dimensional arrays.
Revised: Therefore, we present tropical rational maps as multi-dimensional arrays.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: Given an afﬁne function a : R d 0 → R ; x (cid:55)→ (cid:80) k w k x k + w 0 , we will call the vector ( w 0 , w 1 , . . .
Revised: Given an afﬁne function a : R d 0 → R ; x (cid:55)→ (cid:80) k w k x k + w 0 , we will call the vector ( w 0 , w 1 , . . .

Original: , w d 0 ) its coefﬁcient vector , the scalar w 

Revised: , w d 0 ) its coefﬁcient vector , the scalar w

Original: its constant part and the vector ( w 1 , . .
Revised: its constant part and the vector ( w 1 , . .

Original: . , w d 0 ) its variable part .
Revised: . , w d 0 ) its variable part .

Original: We can represent functions
Revised: We can represent functions

Original: N i : R d 0 → R as in equation (1) in the following way: Let the rows of the matrix
Revised: N i : R d 0 → R as in equation (1) in the following way: Let the rows of the matrix

Original: A + i ∈ R
Revised: A + i ∈ R

Original: n × d 0 and the vector a + i ∈ R n × 1 be the variable and the constant parts of the afﬁne functions a + ij , respectively.
Revised: n × d 0 and the vector a + i ∈ R n × 1 be the variable and the constant parts of the afﬁne functions a + ij , respectively.

Original: (Analogously for A − i and a − i .)
Revised: (Analogously for A − i and a − i .)

Original: We can then deﬁne ( A
Revised: We can then deﬁne ( A

Original: + i , a + i )( x ) = max { A
Revised: + i , a + i )( x ) = max { A

Original: + i x + a + i } , where the maximum is taken over the rows of the resulting column vector.
Revised: + i x + a + i } , where the maximum is taken over the rows of the resulting column vector.

Original: If we deﬁne the formal quotient 1 of matrix-vector pairs by ( A + i , a + i ) /
Revised: If we deﬁne the formal quotient 1 of matrix-vector pairs by ( A + i , a + i ) /

Original: ( A − i , a − i )( x ) = max { A
Revised: ( A − i , a − i )( x ) = max { A

Original: + i x + a + i } − max { A − i x
Revised: + i x + a + i } − max { A − i x

Original: + a − i } , then N i ( x ) =
Revised: + a − i } , then N i ( x ) =

Original: ( A + i , a + i ) /
Revised: ( A + i , a + i ) /

Original: ( A − i , a − i )( x ) , giving us a matrixrepresentation of the function N i .
Revised: ( A − i , a − i )( x ) , giving us a matrixrepresentation of the function N i .

Original: An entire network function with s output dimensions can then be represented by a list (cid:0)
Revised: An entire network function with s output dimensions can then be represented by a list (cid:0)

Original: ( A + i , a + i ) /
Revised: ( A + i , a + i ) /

Original: ( A − i , a − i ) (cid:1) 1 ≤ i ≤ s .
Revised: ( A − i , a − i ) (cid:1) 1 ≤ i ≤ s .

Original: The advantage of the proposed matrix representation of tropical rational maps are natural operations performing calculations that arise for (concatenations of) layers of neural networks (see supplements).
Revised: The advantage of the proposed matrix representation of tropical rational maps are natural operations performing calculations that arise for (concatenations of) layers of neural networks (see supplements).

Original: A dense layer (cid:96) : R d 1 → R d 2 without ReLU activation is represented as a list (cid:0) (
Revised: A dense layer (cid:96) : R d 1 → R d 2 without ReLU activation is represented as a list (cid:0) (

Original: A + i , a + i ) /
Revised: A + i , a + i ) /

Original: ( A − i , a − i ) (cid:1) 1 ≤ i ≤ d 2 .
Revised: ( A − i , a − i ) (cid:1) 1 ≤ i ≤ d 2 .

Original: Denoting by W pos and W neg the positive and negative part of a matrix W , respectively, i.e. w pos ij = max { w ij , 0 } and w neg ij = max {− w ij , 0 } , the matrix representation of a single neuron n i ( x ) = max { w · x + b, 0 } is given by
Revised: Denoting by W pos and W neg the positive and negative part of a matrix W , respectively, i.e. w pos ij = max { w ij , 0 } and w neg ij = max {− w ij , 0 } , the matrix representation of a single neuron n i ( x ) = max { w · x + b, 0 } is given by

Original: [Equation]
Revised: [Equation]

Original: 3. E XTRACTING L INEAR T ERMS OF A C LASSIFICATION N ETWORK
Revised: 3. E XTRACTING L INEAR T ERMS OF A C LASSIFICATION N ETWORK

Original: We now consider a classiﬁcation neural network N with s labels.
Revised: We now consider a classiﬁcation neural network N with s labels.

Original: We show that we can represent the network N with a matrix-vector pair ( A − , a − ) in the denominator that is constant over all output dimensions.
Revised: We show that we can represent the network N with a matrix-vector pair ( A − , a − ) in the denominator that is constant over all output dimensions.

Original: The proof is given in Section C.2 of the supplementary material.
Revised: The proof is given in Section C.2 of the supplementary material.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: Let N : R d → R s be the function of a ReLU neural network for classiﬁcation with s output neurons.
Revised: Let N : R d → R s be the function of a ReLU neural network for classiﬁcation with s output neurons.

Original: Then there are afﬁne functions a + ij , a − j such that
Revised: Then there are afﬁne functions a + ij , a − j such that

Original: [Equation]
Revised: [Equation]

Original: where the maxima of the a − i ( x ) on the right is subtracted from each entry of the vector on the left.
Revised: where the maxima of the a − i ( x ) on the right is subtracted from each entry of the vector on the left.

Original: In terms of our matrices the classiﬁcation network N with s labels can be represented by a list ( ( A + i , a + i ) /
Revised: In terms of our matrices the classiﬁcation network N with s labels can be represented by a list ( ( A + i , a + i ) /

Original: ( A − , a − ) )
Revised: ( A − , a − ) )

Original: 1 ≤ i ≤ s of matrix-vector pairs.
Revised: 1 ≤ i ≤ s of matrix-vector pairs.

Original: The label is then given by argmax 1 ≤ i ≤ s ( A + i , a + i )( x ) .
Revised: The label is then given by argmax 1 ≤ i ≤ s ( A + i , a + i [Equation] ( x ) .

Original: [Table]
Revised: [Table]

Original: set X
Revised: set X

Original: = { ( x i k , i ) }
Revised: = { ( x i k , i ) }

Original: [Table]
Revised: [Table]

Original: D i points
Revised: D i points

Original: [Table]
Revised: [Table]

Original: label
Revised: label

Original: [Table]
Revised: [Table]

Original: Output : Extracted Function N ( X ) =
Revised: Output : Extracted Function N ( X ) =

Original: ( ( A + i , a + i ) /
Revised: ( ( A + i , a + i ) /

Original: ( A − , a − ) )
Revised: ( A − , a − ) )

Original: 1 ≤ i ≤ s 1: W , b ← weight matrix and bias vector of last layer (cid:96)
Revised: 1 ≤ i ≤ s 1: W , b ← weight matrix and bias vector of last layer (cid:96)

Original: [Table]
Revised: [Table]

Original: C − , c − ← column sums of W neg ,b neg
Revised: C − , c − ← column sums of W neg ,

Original: [Table]
Revised: [Table]

Revised: neg

Original: [Table]
Revised: [Table]

Original: i=
Revised: i

Original: [Table]
Revised: [Table]

Original: to s do
Revised: to s do

Original: [Table]
Revised: [Table]

Original: ← rep
Revised: ← rep

Original: [Table]
Revised: [Table]

Original: C+ i
Revised: C

Original: [Table]
Revised: [Table]

Original: D i ) ,
Revised: D [Table]

Original: [Table]
Revised: [Table]

Original: ← rep
Revised: ← rep

Original: [Table]
Revised: [Table]

Original: c
Revised: c

Original: + i
Revised: + i

Original: [Table]
Revised: [Table]

Original: D i )
Revised: D i )

Original: [Table]
Revised: [Table]

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: rows.
Revised: rows.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: a ← a + Ab a ← a + Ab
Revised: a ← a + Ab a ← a + Ab

Original: A ← K + ConvTrans ( A , F )
Revised: A ← K + ConvTrans ( A , F )

Original: A ← K + AW
Revised: A ← K + AW

Original: Flatten
Revised: Flatten

Original: A , A max ← reshape to input shape of (cid:96)
Revised: A , A max ← reshape to input shape of (cid:96)

Original: L-ReLU α ← Leaky ReLU parameter
Revised: L-ReLU α ← Leaky ReLU parameter

Original: ReLU
Revised: ReLU

Original: A kj ← 0 if (cid:96) ( x k ) j
Revised: A kj ← 0 if (cid:96) ( x k ) j

Original: =
Revised: =

Original: A kj ← α · A kj if (cid:96) ( x k ) j ≤
Revised: A kj ← α · A kj if (cid:96) ( x k ) j ≤

Original: How to get the extracted function N ( X ) of (2) from the network N . TropEx extracts, for each data point x k of label
Revised: How to get the extracted function N ( X ) of (2) from the network N . TropEx extracts, for each data point x k of label

Original: i , the afﬁne functions a + ij and a − l from the network representation in (3) such that a + ij ( x k ) ≥ a +˜ i ˜
Revised: i , the afﬁne functions a + ij and a − l from the network representation in (3) such that a + ij ( x k ) ≥ a +˜ i ˜

Original: j ( x k ) , a − l ( x k ) ≥ a ˜ l ( x k ) for all ˜ i, ˜ j, ˜ l .
Revised: j ( x k ) , a − l ( x k ) ≥ a ˜ l ( x k ) for all ˜ i, ˜ j, ˜ l .

Original: We start with the last layer of the network and inductively merge new layers into the existing matrix pairs.
Revised: We start with the last layer of the network and inductively merge new layers into the existing matrix pairs.

Original: The merge operations depend on the type of the layer as shown in Table 1.
Revised: The merge operations depend on the type of the layer as shown in Table 1.

Original: Putting things together gives Algorithm 3.1, which has the extracted function N ( X ) as its output.
Revised: Putting things together gives Algorithm 3.1, which has the extracted function N ( X ) as its output.

Original: The run-time and storage complexities  point correspond to 3 forward passes through the network.
Revised: The run-time and storage complexities data point correspond to 3 forward passes through the network.

Original: Theorem 3.3 states that TropEx indeed results in a selection of linear terms based on a data set of points and that the extracted tropical function agrees with the network on neighbourhoods of all these points.
Revised: Theorem 3.3 states that TropEx indeed results in a selection of linear terms based on a data set of points and that the extracted tropical function agrees with the network on neighbourhoods of all these points.

Original: Its proof and the complete, nontrivial derivation of the algorithm are in the appendix, where we develop a framework that enables calculations on tropical matrices that correspond to manipulations of tropical functions they represent.
Revised: Its proof and the complete, nontrivial derivation of the algorithm are in the appendix, where we develop a framework that enables calculations on tropical matrices that correspond to manipulations of tropical functions they represent.

Original: For illustrative purposes, we also present a worked-out example of applying TropEx to a toy neural network there.
Revised: For illustrative purposes, we also present a worked-out example of applying TropEx to a toy neural network there.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Let N =
Revised: Let N =

Original: ( N 1 , . .
Revised: ( N 1 , . .

Original: . , N s ) :
Revised: . , N s ) :

Original: R d → R s be the function of a ReLU neural network for classiﬁcation into s classes.
Revised: R d → R s be the function of a ReLU neural network for classiﬁcation into s classes.

Original: Let N ( X ) be the network obtained from Algorithm 3.1, applied to N using a data set X
Revised: Let N ( X ) be the network obtained from Algorithm 3.1, applied to N using a data set X

Original: = { ( xi k  , i ) | 1 ≤ i ≤ s, 1 ≤ i k ≤ D i } of D i data points xi k  given label i by N .
Revised: = { ( x k j , i ) | 1 ≤ i ≤ s, 1 ≤ j ≤ D i } of D i data points x k j given label i by N .

Original: (There are D points  in total).
Revised: (There are D points x 1 , . . . , x D in total).

Original: Then, (1) for all labels i , N ( X ) i [Equation] max { a 
Revised: Then, (1) for all labels i , N ( X ) i ( x ) = max { a +

Original: [Equation]

Original:  ( x ) , . . .
Revised: i 1 ( x ) , . . .

Original: , a  − D ( x ) } , where the a + ij and a − j are extracted from a representation of N i as in Equation 3; and (2) every data point x k has a neighbourhood U k on which the maximum of the extracted function agrees with the maximal network output:
Revised: , a + iD i ( x ) } − max { a − 1 ( x ) , . . . , a − D ( x ) } , where the a + ij and a − j are extracted from a representation of N i as in Equation 3; and (2) every data point x k has a neighbourhood U k on which the maximum of the extracted function agrees with the maximal network output:

Original: [Equation]
Revised: [Equation]

Original: In particular, N ( X ) and N classify all points in U k by assigning the same label.
Revised: In particular, N ( X ) and N classify all points in U k by assigning the same label.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: TropEx extracts a function containing only linear terms corresponding to regions on which the given data lies.
Revised: TropEx extracts a function containing only linear terms corresponding to regions on which the given data lies.

Original: The extracted function agrees with the network on this data.
Revised: The extracted function agrees with the network on this data.

Original: This allows us to compare linear regions of train and test data, to separate the network structure from the information contained in the linear coefﬁcients, and to test how well the linear coefﬁcients generalize to test data.
Revised: This allows us to compare linear regions of train and test data, to separate the network structure from the information contained in the linear coefﬁcients, and to test how well the linear coefﬁcients generalize to test data.

Original: Setup We train neural networks on MNIST (LeCun et al., 2010) and CIFAR10 (Krizhevsky, 2009).
Revised: Setup We train neural networks on MNIST (LeCun et al., 2010) and CIFAR10 (Krizhevsky, 2009).

Original: After training, we use training data points x ( tr ) to extract linear terms a + x ( tr )  and a − x ( tr )  and construct an extracted function N ( X ) as in equation (2).
Revised: After training, we use training data points x ( tr ) to extract linear terms a + x ( tr ) ( x ) and a − x ( tr ) ( x ) and construct an extracted function N ( X ) as in equation (2).

Original: For some experiments, we also extract linear terms a + x ( te )  and a − x ( te )  corresponding to test data points x ( te ) .
Revised: For some experiments, we also extract linear terms a + x ( te ) ( x ) and a − x ( te ) ( x ) corresponding to test data points x ( te ) .

Original: Regarding the architecture, we use fullyconnected networks, AllCNN-C from Springenberg et al.
Revised: Regarding the architecture, we use fully-connected networks, AllCNN-C from Springenberg et al.

Original: and variations of VGG-B from Simonyan & Zisserman (2014).
Revised: and variations of VGG-B from Simonyan & Zisserman (2014).

Original: Section D in the appendix summarizes all architectures we used in our experiments.
Revised: Section D in the appendix summarizes all architectures we used in our experiments.

Original: If not stated otherwise, we use architectures Conv for CNNs and FCN8 for FCNs in our experiments.
Revised: If not stated otherwise, we use architectures Conv for CNNs and FCN8 for FCNs in our experiments.

Original: It is not our goal to train networks to state of the art-performance, but rather to compare variations of simple networks which are composed of the layers shown in Table 1.
Revised: It is not our goal to train networks to state of the art-performance, but rather to compare variations of simple networks which are composed of the layers shown in Table 1.

Original: All layers have ReLU activations except for the last layer where we apply a softmax output function into the ten respective classes.
Revised: All layers have ReLU activations except for the last layer where we apply a softmax output function into the ten respective classes.

Original: 3 We train ﬁve networks of each architecture to ensure the consistency of our results.
Revised: 3 We train ﬁve networks of each architecture to ensure the consistency of our results.

Original: Further details on the training setup can be found in the appendix.
Revised: Further details on the training setup can be found in the appendix.

Original: Counting of Linear Regions At ﬁrst, we investigate how training and test samples are distributed over the linear regions of the neural networks.
Revised: Counting of Linear Regions At ﬁrst, we investigate how training and test samples are distributed over the linear regions of the neural networks.

Original: For each data point x , let a x = a + x − a − x be network function corresponding to the linear region on which x lies.
Revised: For each data point x , let a x = a + x − a − x be network function corresponding to the linear region on which x lies.

Original: Since a x (cid:54) = a x (cid:48) for all training and test points x , x (cid:48) , we see that all points lie in different linear regions.
Revised: Since a x (cid:54) = a x (cid:48) for all training and test points x , x (cid:48) , we see that all points lie in different linear regions.

Original: This is not a result of overﬁtting during training: All data points also occupy different regions when we count the number of linear regions after 1, 3, 5, 10, 15, 20, 30, 40 and 50 epochs of training on CIFAR10, and from epochs 1 to 20 on MNIST.
Revised: This is not a result of overﬁtting during training: All data points also occupy different regions when we count the number of linear regions after 1, 3, 5, 10, 15, 20, 30, 40 and 50 epochs of training on CIFAR10, and from epochs 1 to 20 on MNIST.

Original: Therefore we conclude that generalization capabilities of neural networks cannot be explained by test samples falling into the same linear region as training samples (or, in other words, by test samples inducing exactly the same activation pattern as training samples).
Revised: Therefore we conclude that generalization capabilities of neural networks cannot be explained by test samples falling into the same linear region as training samples (or, in other words, by test samples inducing exactly the same activation pattern as training samples).

Original: Examining Function Coefﬁcients With test samples falling into different regions than training samples, it is conceivable that neighboring regions could still be “nearly identical” and test samples would fall into such neighboring regions of training samples.
Revised: Examining Function Coefﬁcients With test samples falling into different regions than training samples, it is conceivable that neighboring regions could still be “nearly identical” and test samples would fall into such neighboring regions of training samples.

Original: To clarify, we ran experiments to test similarity of linear regions for test samples before and after reduction.
Revised: To clarify, we ran experiments to test similarity of linear regions for test samples before and after reduction.

Original: We take a test sample and extract the linear coefﬁcient of its linear region in both the original network N and in the extracted function N ( X ) .
Revised: We take a test sample and extract the linear coefﬁcient of its linear region in both the original network N and in the extracted function N ( X ) .

Original: For these linear coefﬁcient vectors, we calculate the (i) angle and (ii) Euclidean norm difference.
Revised: For these linear coefﬁcient vectors, we calculate the (i) angle and (ii) Euclidean norm difference.

Original: Figure 2 shows both values for all test samples, where we differentiate between those test points x ( te ) that get correctly classiﬁed by N ( X ) (in blue) and those that get a wrong label (in red).
Revised: Figure 2 shows both values for all test samples, where we differentiate between those test points x ( te ) that get correctly classiﬁed by N ( X ) (in blue) and those that get a wrong label (in red).

Original: We observe a clear difference between the CNN and the FCN.
Revised: We observe a clear difference between the CNN and the FCN.

Original: For CNNs, the coefﬁcient vectors of both training and test afﬁne functions are all close to orthogonal for correctly as well as incorrectly classiﬁed points, For FCNs, angle and distance of correctly classiﬁed points are smaller than for incorrectly classiﬁed ones, but still far away from zero and therefore rule out the possibility of test samples falling into very similar neighboring regions.
Revised: For CNNs, the coefﬁcient vectors of both training and test afﬁne functions are all close to orthogonal for correctly as well as incorrectly classiﬁed points, For FCNs, angle and distance of correctly classiﬁed points are smaller than for incorrectly classiﬁed ones, but still far away from zero and therefore rule out the possibility of test samples falling into very similar neighboring regions.

Original: Finally, instead of comparing linear coefﬁcients, we also tested the similarity in activation patterns before and after extraction.
Revised: Finally, instead of comparing linear coefﬁcients, we also tested the similarity in activation patterns before and after extraction.

Original: The results in section E.4 of the appendix show that in each layer approx 80% of neuron activations agree between test and training region, showing that also the activation patterns of test samples deviate considerably and generalization cannot be explained by very similar activation patterns.
Revised: The results in section E.5 of the appendix show that in each layer approx 80% of neuron activations agree between test and training region, showing that also the activation patterns of test samples deviate considerably and generalization cannot be explained by very similar activation patterns.

Original: Accuracy of the Extracted Functions As predicted by Theorem 3.3, the maximum of the extracted function N ( X ) agrees with the maximum of the original network on all training points for each of our networks.
Revised: Accuracy of the Extracted Functions As predicted by Theorem 3.3, the maximum of the extracted function N ( X ) agrees with the maximum of the original network on all training points for each of our networks.

Original: In particular, network and extracted function assign the same label to each training point.
Revised: In particular, network and extracted function assign the same label to each training point.

Original: To investigate how well the coefﬁcients of these linear regions generalize to unseen data, we compare test accuracy of network and extracted function in Table 2.
Revised: To investigate how well the coefﬁcients of these linear regions generalize to unseen data, we compare test accuracy of network and extracted function in Table 2.

Original: We see a consistent difference between
Revised: We see a consistent difference between

Original: [Table]
Revised: [Table]

Original: CNNs and FCNs across both data sets and all architectures: There is a drastic drop in the test accuracy of the CNNs, as opposed to a relatively small drop in the accuracy of the FCNs.
Revised: CNNs and FCNs across both data sets and all architectures: There is a drastic drop in the test accuracy of the CNNs, as opposed to a relatively small drop in the accuracy of the FCNs.

Original: Interestingly, for
Revised: Interestingly, for

Original: MNIST, the extracted tropical function has almost the same test accuracy as the original network.
Revised: MNIST, the extracted tropical function has almost the same test accuracy as the original network.

Original: This is surprising as all known bounds on the number of linear regions of the original network suggest numbers of the order of 10 80 up to over 10 17000 from which we only observe 60.000 after reduction.
Revised: This is surprising as all known bounds on the number of linear regions of the original network suggest numbers of the order of 10 80 up to over 10 17000 from which we only observe 60.000 after reduction.

Original: Hence, for fully connected networks on a simple task, the coefﬁcients used on training data generalize well to test data
Revised: Hence, for fully connected networks on a simple task, the coefﬁcients used on training data generalize well to test data

Original: [Table]
Revised: [Table]

Original: but for complex data, the learned coefﬁcients generalize worse.
Revised: but for complex data, the learned coefﬁcients generalize worse.

Original: This is remarkable, since previous studies (Hanin & Rolnick, 2019b;a; Zhang & Wu, 2020) of linear regions were forced to base their experiments on small data sets (or small networks) for computational reasons
Revised: This is remarkable, since previous studies (Hanin & Rolnick, 2019b;a; Zhang & Wu, 2020) of linear regions were forced to base their experiments on small data sets (or small networks) for computational reasons

Original: [Table]
Revised: [Table]

Original: and it seems that care must be taken when generalizing observations to more complex tasks.
Revised: and it seems that care must be taken when generalizing observations to more complex tasks.

Original: Moreover, the results reveal another difference between FCNs and CNNs that we further investigate.
Revised: Moreover, the results reveal another difference between FCNs and CNNs that we further investigate.

Original: Transformation Process Figure 3, left shows how test accuracy and agreement with the original network develop during the transformation from full network N to its extracted form N ( X ) .
Revised: Transformation Process Figure 3, left shows how test accuracy and agreement with the original network develop during the transformation from full network N to its extracted form N ( X ) .

Original: Starting with the last layer, TropEx iteratively merges layers into an extracted function.
Revised: Starting with the last layer, TropEx iteratively merges layers into an extracted function.

Original: The x-axis shows which layers have been merged to an extracted tropical function at each step.
Revised: The x-axis shows which layers have been merged to an extracted tropical function at each step.

Original: The graph shows accuracy values for passing test samples through the original network until the layer that represents the input to the extracted tropical function and then applying this extracted function.
Revised: The graph shows accuracy values for passing test samples through the original network until the layer that represents the input to the extracted tropical function and then applying this extracted function.

Original: At the right end of the plots all layers have been merged to the tropical function N ( X ) .
Revised: At the right end of the plots all layers have been merged to the tropical function N ( X ) .

Original: There is a clear difference between fully-connected networks and CNNs, which is consistent over all networks (Figure 3, right).
Revised: There is a clear difference between fully-connected networks and CNNs, which is consistent over all networks (Figure 3, right).

Original: The fully connected network Wide and the convolutional network Narrow have exactly the same number of nodes after each parameter layer, but the drop in test accuracy for Narrow is almost 5 times the drop for Wide (36.6% vs 8.1%).
Revised: The fully connected network Wide and the convolutional network Narrow have exactly the same number of nodes after each parameter layer, but the drop in test accuracy for Narrow is almost 5 times the drop for Wide (36.6% vs 8.1%).

Original: Starting with a much superior performance, after reduction of linear terms the convolutional network is outperformed by the fully connected network.
Revised: Starting with a much superior performance, after reduction of linear terms the convolutional network is outperformed by the fully connected network.

Original: Since Narrow has only few connections between its nodes and Wide is fully-connected, it would be reasonable to assume that the number of linear regions of Wide is much greater than the number of linear regions of
Revised: Since Narrow has only few connections between its nodes and Wide is fully-connected, it would be reasonable to assume that the number of linear regions of Wide is much greater than the number of linear regions of

Original: Narrow , since its theoretical upper bound is easily seen to be higher.
Revised: Narrow , since its theoretical upper bound is easily seen to be higher.

Original: Hence, it would be expected that an extraction of a ﬁxed number of linear terms resulted in a smaller change of results for Narrow and the results were reversed.
Revised: Hence, it would be expected that an extraction of a ﬁxed number of linear terms resulted in a smaller change of results for Narrow and the results were reversed.

Original: This contradicts our intuition about how CNNs and FCNs work from the perspective of network expressivity in terms of the number of linear regions.
Revised: This contradicts our intuition about how CNNs and FCNs work from the perspective of network expressivity in terms of the number of linear regions.

Original: Network Training We compare the performance of extracted functions N ( X ) during training of the network N .
Revised: Network Training We compare the performance of extracted functions N ( X ) during training of the network N .

Original: Figure 4 displays the test accuracy (mean and standard deviation over 5 networks) of the extracted function and the agreement of label assignments of extracted form with the original network function for Narrow (CNN) and Wide (fully connected) for several epochs.
Revised: Figure 4 displays the test accuracy (mean and standard deviation over 5 networks) of the extracted function and the agreement of label assignments of extracted form with the original network function for Narrow (CNN) and Wide (fully connected) for several epochs.

Original: The difference between fully connected and convolutional networks is here even more striking.
Revised: The difference between fully connected and convolutional networks is here even more striking.

Original: For the CNN, the agreement between extracted tropical function and original network function falls rapidly after only one epoch and only slightly reduces from there.
Revised: For the CNN, the agreement between extracted tropical function and original network function falls rapidly after only one epoch and only slightly reduces from there.

Original: For the FCN, the agreement decreases slowly over the entire 50 training epochs and it never reaches a value as low as the CNN after its ﬁrst epoch.
Revised: For the FCN, the agreement decreases slowly over the entire 50 training epochs and it never reaches a value as low as the CNN after its ﬁrst epoch.

Original: Information encoded in linear coefﬁcients The extracted functions all share the number of linear terms, hence their difference in performance must be explained by the coefﬁcient values.
Revised: Information encoded in linear coefﬁcients The extracted functions all share the number of linear terms, hence their difference in performance must be explained by the coefﬁcient values.

Original: With this in mind, we attempt an interpretation of the above results and hypothesize that the differences lies in how FCN and CNN store important information for the classiﬁcation task in the linear coefﬁcients of linear regions.
Revised: With this in mind, we attempt an interpretation of the above results and hypothesize that the differences lies in how FCN and CNN store important information for the classiﬁcation task in the linear coefﬁcients of linear regions.

Original: An FCN has the full freedom to compose weights to tailored coefﬁcients of linear regions, whereas CNNs impose a structure on the weight space by ﬁlters and weight sharing, which results in the incapability to compose tailored linear coefﬁcients for correct label assignments.
Revised: An FCN has the full freedom to compose weights to tailored coefﬁcients of linear regions, whereas CNNs impose a structure on the weight space by ﬁlters and weight sharing, which results in the incapability to compose tailored linear coefﬁcients for correct label assignments.

Original: Instead, the structural properties of convolutions composing the network function play a signiﬁcant role for generalization.
Revised: Instead, the structural properties of convolutions composing the network function play a signiﬁcant role for generalization.

Original: By extracting linear terms we remove the structure, which strongly changes the outcome on test data as the coefﬁcients of linear regions alone are limited in meaning.
Revised: By extracting linear terms we remove the structure, which strongly changes the outcome on test data as the coefﬁcients of linear regions alone are limited in meaning.

Original: As training progresses, to achieve higher accuracy, the FCN reduces the information stored in linear coefﬁcients and also learns to use some structure, so that the removal of this structure could explain the decrease in performance of the extracted function.
Revised: As training progresses, to achieve higher accuracy, the FCN reduces the information stored in linear coefﬁcients and also learns to use some structure, so that the removal of this structure could explain the decrease in performance of the extracted function.

Original: An experiment, where we visually inspect misclassiﬁed images (Appendix E.1) is in line with this interpretation suggesting that the object shape is encoded in the linear coefﬁcients of the FCNs, but for CNNs only simple features such as background color are encoded in the linear coefﬁcients of linear regions.
Revised: An experiment, where we visually inspect misclassiﬁed images (Appendix E.2) is in line with this interpretation suggesting that the object shape is encoded in the linear coefﬁcients of the FCNs, but for CNNs only simple features such as background color are encoded in the linear coefﬁcients of linear regions.

Original: Re-training the network The observation of smaller angles for FCNs in Figure 2 further supports our interpretation that coefﬁcient values of linear regions play a larger role in classiﬁcation for FCNs, since smaller angles together with small Euclidean distance are explained by a similarity of the coefﬁcient values.
Revised: Re-training the network The observation of smaller angles for FCNs in Figure 2 further supports our interpretation that coefﬁcient values of linear regions play a larger role in classiﬁcation for FCNs, since smaller angles together with small Euclidean distance are explained by a similarity of the coefﬁcient values.

Original: This suggests to also compare the similarity of linear coefﬁcients after re-training the network in order to further understand the information encoded in the linear coefﬁcients.
Revised: This suggests to also compare the similarity of linear coefﬁcients after re-training the network in order to further understand the information encoded in the linear coefﬁcients.

Original: Again,we ﬁnd that the coefﬁcient vectors of two separately trained CNNs are close to orthogonal, whereas both angles and distances are considerably smaller for FCNs.
Revised: Again,we ﬁnd that the coefﬁcient vectors of two separately trained CNNs are close to orthogonal, whereas both angles and distances are considerably smaller for FCNs.

Original: 5 Plots are shown in Appendix E.5.
Revised: 5 Plots are shown in Appendix E.6.

Original: For each dimension we additionally compute the Pearson correlation between the linear coefﬁcients of two separately trained networks over all training samples.
Revised: For each dimension we additionally compute the Pearson correlation between the linear coefﬁcients of two separately trained networks over all training samples.

Original: We reduce the resulting vector to a single number by averaging the correlation factors over the dimensions.
Revised: We reduce the resulting vector to a single number by averaging the correlation factors over the dimensions.

Original: We experiment with two pairs of FCNs and two pairs of CNNs trained on CIFAR10, and one pair of FCNs trained on MNIST.
Revised: We experiment with two pairs of FCNs and two pairs of CNNs trained on CIFAR10, and one pair of FCNs trained on MNIST.

Original: Figure 5 shows the evolution of the correlation during training, conﬁrming that the similarity of coefﬁcient values is also much larger as measured by correlation.
Revised: Figure 5 shows the evolution of the correlation during training, conﬁrming that the similarity of coefﬁcient values is also much larger as measured by correlation.

Original: Interestingly, for the FCN we notice a drastic jump in the correlation values to a larger value precisely when the training accuracy reaches 100% , showing a signiﬁcant correlation of linear coefﬁcients after re-training after convergence.
Revised: Interestingly, for the FCN we notice a drastic jump in the correlation values to a larger value precisely when the training accuracy reaches 100% , showing a signiﬁcant correlation of linear coefﬁcients after re-training after convergence.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The function of a ReLU network is piecewise linear, with an astronomically high number of linear regions.
Revised: The function of a ReLU network is piecewise linear, with an astronomically high number of linear regions.

Original: We introduced TropEx, an algorithm to systematically extract linear regions based on data points.
Revised: We introduced TropEx, an algorithm to systematically extract linear regions based on data points.

Original: The derivation is based on a matrix representation of tropical functions that supports efﬁcient algorithmic development.
Revised: The derivation is based on a matrix representation of tropical functions that supports efﬁcient algorithmic development.

Original: TropEx enables investigations of the linear components of piecewise linear network functions:
Revised: TropEx enables investigations of the linear components of piecewise linear network functions:

Original: By extracting the network’s linear terms, the algorithm allows us to compare training and test regions and to systematically analyze their linear coefﬁcients.
Revised: By extracting the networks’ linear terms, the algorithm allows us to compare training and test regions and to systematically analyze their linear coefﬁcients.

Original: Applying TropEx to fully-connected and convolutional architectures shows signiﬁcant differences between linear regions of CNNs and FCNs.
Revised: Applying TropEx to fully-connected and convolutional architectures shows signiﬁcant differences between linear regions of CNNs and FCNs.

Original: Other possible use cases are outlined in Appendix G.
Revised: Other possible use cases are outlined in Appendix G.

Original: Our ﬁndings indicate a potential beneﬁt from shifting focus from counting linear regions to an understanding of their interplay, as differences between CNNs and FCNs may be found in the coefﬁcients of the extracted linear terms.
Revised: Our ﬁndings indicate a potential beneﬁt from shifting focus from counting linear regions to an understanding of their interplay, as differences between CNNs and FCNs may be found in the coefﬁcients of the extracted linear terms.

Original: Several measures of similarity indicate that the linear terms of CNNs are more diverse than those of FCNs and suggest that CNNs efﬁciently exploit the structure imposed by their architecture, whereas FCNs rely on encoding information in the values of linear coefﬁcients.
Revised: Several measures of similarity indicate that the linear terms of CNNs are more diverse than those of FCNs and suggest that CNNs efﬁciently exploit the structure imposed by their architecture, whereas FCNs rely on encoding information in the values of linear coefﬁcients.