Original: T HE R OLE OF P ERMUTATION I NVARIANCE IN L INEAR M ODE C ONNECTIVITY OF N EURAL N ETWORKS
Revised: T HE R OLE OF P ERMUTATION I NVARIANCE IN L INEAR M ODE C ONNECTIVITY OF N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them.
Revised: In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them.

Original: Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it.
Revised: Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it.

Original: We further provide a preliminary theoretical result to support our conjecture.
Revised: We further provide a preliminary theoretical result to support our conjecture.

Original: Our conjecture has implications for lottery ticket hypothesis, distributed training and ensemble methods.
Revised: Our conjecture has implications for lottery ticket hypothesis, distributed training and ensemble methods.

Original: The source code is available at https://github.com/AnonymousMLSubmission/PermutationInvariance .
Revised: The source code is available at https://github.com/AnonymousMLSubmission/PermutationInvariance .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Understanding the loss landscape of deep neural networks has been the subject of many studies due to its close connections to optimization and generalization (Li et al., 2017; Mei et al., 2018; Geiger et al., 2019; Nguyen et al., 2018;
Revised: Understanding the loss landscape of deep neural networks has been the subject of many studies due to its close connections to optimization and generalization (Li et al., 2017; Mei et al., 2018; Geiger et al., 2019; Nguyen et al., 2018;

Original: Fort et al., 2019; Baldassi et al., 2020).
Revised: Fort et al., 2019; Baldassi et al., 2020).

Original: Empirical observations suggest that loss landscape of deep networks has many minima (Keskar et al., 2017; Draxler et al., 2018; Zhang et al., 2017).
Revised: Empirical observations suggest that loss landscape of deep networks has many minima (Keskar et al., 2017; Draxler et al., 2018; Zhang et al., 2017).

Original: One reason behind the abundance of minima is over-parametrization.
Revised: One reason behind the abundance of minima is over-parametrization.

Original: Over-parametrized networks have enough capacity to present different functions that behave similarly on the training data but vastly different on other inputs (Neyshabur et al., 2017; Nguyen et al., 2018;
Revised: Over-parametrized networks have enough capacity to present different functions that behave similarly on the training data but vastly different on other inputs (Neyshabur et al., 2017; Nguyen et al., 2018;

Original: Li et al., 2018; Liu et al., 2020).
Revised: Li et al., 2018; Liu et al., 2020).

Original: Another contributing factor is the existence of scale and permutation invariances which allows the same function to be represented with many different parameter values of the same network and imposes a counter-intuitive geometry on the loss landscape (Neyshabur et al., 2015;
Revised: Another contributing factor is the existence of scale and permutation invariances which allows the same function to be represented with many different parameter values of the same network and imposes a counter-intuitive geometry on the loss landscape (Neyshabur et al., 2015;

Original: Brea et al., 2019a).
Revised: Brea et al., 2019a).

Original: Previous work study the relationship between different minima found by SGD and establish that they are connected by a path of non-increasing loss; however, they are not connected by a linear path (Freeman & Bruna, 2016; Draxler et al., 2018; Garipov et al., 2018).
Revised: Previous work study the relationship between different minima found by SGD and establish that they are connected by a path of non-increasing loss; however, they are not connected by a linear path (Freeman & Bruna, 2016; Draxler et al., 2018; Garipov et al., 2018).

Original: This phenomenon is often referred to as mode connectivity (Garipov et al., 2018) and the loss increase on the path between two solutions is often referred to as (energy) barrier (Draxler et al., 2018).
Revised: This phenomenon is often referred to as mode connectivity (Garipov et al., 2018) and the loss increase on the path between two solutions is often referred to as (energy) barrier (Draxler et al., 2018).

Original: Understanding linear mode connectivity (LMC) is highly motivated by several direct conceptual and practical implications from pruning and sparse training to distributed optimization and ensemble methods.
Revised: Understanding linear mode connectivity (LMC) is highly motivated by several direct conceptual and practical implications from pruning and sparse training to distributed optimization and ensemble methods.

Original: The relationship between LMC and pruning was established by Frankle et al.
Revised: The relationship between LMC and pruning was established by Frankle et al.

Original: (2020) where they showed the correspondence between LMC and the well-known lottery ticket hypothesis (LTH)
Revised: (2020) where they showed the correspondence between LMC and the well-known lottery ticket hypothesis (LTH)

Original: (Frankle & Carbin, 2019).
Revised: (Frankle & Carbin, 2019).

Original: In short, LTH conjectures that neural networks contain sparse subnetworks that can be trained in isolation, from initialization, or early in training to achieve comparable test accuracy.
Revised: In short, LTH conjectures that neural networks contain sparse subnetworks that can be trained in isolation, from initialization, or early in training to achieve comparable test accuracy.

Original: Frankle et al.
Revised: Frankle et al.

Original: (2020) showed that solutions that are linearly connected with no barrier have the same lottery ticket.
Revised: (2020) showed that solutions that are linearly connected with no barrier have the same lottery ticket.

Original: They further discuss how linear-connectivity is associated with stability of SGD.
Revised: They further discuss how linear-connectivity is associated with stability of SGD.

Original: This view suggests that SGD solutions that are linearly connected with no barrier can be thought of as being in the same basin of the loss landscape and once SGD converges to a basin, it shows a stable behavior inside the basin 1 .
Revised: This view suggests that SGD solutions that are linearly connected with no barrier can be thought of as being in the same basin of the loss landscape and once SGD converges to a basin, it shows a stable behavior inside the basin 1 .

Original: Because of the direct correspondence between LMC and
Revised: Because of the direct correspondence between LMC and

Original: LTH, any understanding of LMC, has implications for LTH, stability of SGD and pruning techniques.
Revised: LTH, any understanding of LMC, has implications for LTH, stability of SGD and pruning techniques.

Original: Linear mode connectivity has also direct implications for ensemble methods and distributed training.
Revised: Linear mode connectivity has also direct implications for ensemble methods and distributed training.

Original: Ensemble methods highly depend on an understanding of the loss landscape and being able to sample from solutions.
Revised: Ensemble methods highly depend on an understanding of the loss landscape and being able to sample from solutions.

Original: Better understanding of mode connectivity has been shown to be essential in devising better ensemble methods (Garipov et al., 2018).
Revised: Better understanding of mode connectivity has been shown to be essential in devising better ensemble methods (Garipov et al., 2018).

Original: Linear mode connectivity between solutions or 
Revised: Linear mode connectivity between solutions or checkpoints also allows for weight averaging techniques for distributed optimization to be used as effectively in deep learning as convex optimization (Scaman et al., 2019).

Original: [Figure]
Revised: [Figure]

Original:  a rr i e r ( O u r
Revised: B a rr i e r ( O u r

Original: M o d e l )
Revised: M o d e l )

Original: [Figure]
Revised: [Figure]

Original: checkpoints also allows for weight averaging techniques for distributed optimization to be used as effectively in deep learning as convex optimization (Scaman et al., 2019)

Original: In this paper, we conjecture that by taking permutation invariance into account, the loss landscape can be simpliﬁed signiﬁcantly resulting in linear mode connectivity between SGD solutions.
Revised: In this paper we conjecture that by taking permutation invariance into account, the loss landscape can be simpliﬁed signiﬁcantly resulting in linear mode connectivity between SGD solutions.

Original: We investigate this conjecture both theoretically and empirically through extensive experiments.
Revised: We investigate this conjecture both theoretically and empirically through extensive experiments.

Original: We show how our attempts fall short of refuting this hypothesis and end up as supporting evidence for it (see Figure 1).
Revised: We show how our attempts fall short of refuting this hypothesis and end up as supporting evidence for it (see Figure 1).

Original: We believe our conjecture sheds light into the structure of loss landscape and could lead to practical implications for the aforementioned areas.
Revised: We believe our conjecture sheds light into the structure of loss landscape and could lead to practical implications for the aforementioned areas.

Original: Contributions .
Revised: Contributions .

Original: This paper makes the following contributions:
Revised: This paper makes the following contributions:

Original: • We study linear mode connectivity (LMC) between solutions trained from different initializations and investigate how it is affected by choices such as width, depth and task difﬁculty for fully connected and convolutional networks ( Section 2).
Revised: • We study linear mode connectivity (LMC) between solutions trained from different initializations and investigate how it is affected by choices such as width, depth and task difﬁculty for fully connected and convolutional networks ( Section 2).

Original: • We introduce our main conjecture in Section 3: If invariances are taken into account, there will likely be no barrier on the linear interpolation of SGD solutions (see the left panel of Figure 1).
Revised: • We introduce our main conjecture in Section 3: If invariances are taken into account, there will likely be no barrier on the linear interpolation of SGD solutions (see the left panel of Figure 1).

Original: • By investigating the conjecture theoretically, we prove that it holds for a wide enough fully-connected network with one hidden layer at random initialization ( Section 3).
Revised: • By investigating the conjecture theoretically, we prove that it holds for a wide enough fully-connected network with one hidden layer at random initialization ( Section 3).

Original: • In Section 4, we provide strong empirical evidence in support of our conjecture.
Revised: • In Section 4, we provide strong empirical evidence in support of our conjecture.

Original: To overcome the computational challenge of directly evaluating the hypothesis empirically, which requires searching in the space of all possible permutations, we propose an alternative approach.
Revised: To overcome the computational challenge of directly evaluating the hypothesis empirically, which requires searching in the space of all possible permutations, we propose an alternative approach.

Original: We consider a set of solutions corresponding to random permutations of a single ﬁxed SGD solution (our model) and show several empirical evidences suggesting our model is a good approximation for all SGD solutions(real world) with different random seeds (see the middle and right panel of Figure 1).
Revised: We consider a set of solutions corresponding to random permutations of a single ﬁxed SGD solution (our model) and show several empirical evidences suggesting our model is a good approximation for all SGD solutions(real world) with different random seeds (see the middle and right panel of Figure 1).

Original: Further related work .
Revised: Further related work .

Original: Permutation symmetry of neurons in every layer results in multiple equivalent minima connected via saddle points.
Revised: Permutation symmetry of neurons in every layer results in multiple equivalent minima connected via saddle points.

Original: Few studies investigate the role of these symmetries in the context of connectivity of different basins.
Revised: Few studies investigate the role of these symmetries in the context of connectivity of different basins.

Original: Given a network with L layers of minimal widths r ∗ 1 , ..., r ∗ L − 1 that reaches zero-loss minima at r 1 ! , ..., r L − 1 !
Revised: Given a network with L layers of minimal widths r ∗ 1 , ..., r ∗ L − 1 that reaches zero-loss minima at r 1 ! , ..., r L − 1 !

Original: isolated points (permutations of one another), ¸Sim¸sek et al.
Revised: isolated points (permutations of one another), ¸Sim¸sek et al.

Original: (2021) showed that adding one extra neuron to each layer is sufﬁcient to connect all these previously discrete minima into a single manifold.
Revised: (2021) showed that adding one extra neuron to each layer is sufﬁcient to connect all these previously discrete minima into a single manifold.

Original: Fukumizu & Amari (2000) prove that a point corresponding to the global minimum of a smaller model can be a local minimum or a saddle point of the larger model.
Revised: Fukumizu & Amari (2000) prove that a point corresponding to the global minimum of a smaller model can be a local minimum or a saddle point of the larger model.

Original: Brea et al.
Revised: Brea et al.

Original: (2019b)
Revised: (2019b)

Original: ﬁnd smooth paths between equivalent global minima that lead through a permutation point, i.e. , where the input and output weight vectors of two neurons in the same hidden layer interchange.
Revised: ﬁnd smooth paths between equivalent global minima that lead through a permutation point, i.e. , where the input and output weight vectors of two neurons in the same hidden layer interchange.

Original: They describe a method to permute all neuron indices in the same layer at the same cost.
Revised: They describe a method to permute all neuron indices in the same layer at the same cost.

Original: Tatro et al.
Revised: Tatro et al.

Original: (2020) showed that aligning the neurons in two different neural networks makes it easier to ﬁnd second order curves between them in the loss landscape where barriers are absent.
Revised: (2020) showed that aligning the neurons in two different neural networks makes it easier to ﬁnd second order curves between them in the loss landscape where barriers are absent.

Original:  L OSS B ARRIERS
Revised:  L OSS B ARRIERS

Original: In this section, we ﬁrst give a formal deﬁnition for linear mode connectivity and study how it is affected by different factors such as network width, depth, and task difﬁculty for a variety of architectures.
Revised: In this section, we ﬁrst give a formal deﬁnition for linear mode connectivity and study how it is affected by different factors such as network width, depth, and task difﬁculty for a variety of architectures.

Original: 2. D EFINITIONS
Revised: 2. D EFINITIONS

Original: Let f θ ( · ) be a function presented by a neural network with parameter vector θ that includes all parameters and L ( θ ) be the any given loss (e.g., train or test error) of f θ ( · ) .
Revised: Let f θ ( · ) be a function presented by a neural network with parameter vector θ that includes all parameters and L ( θ ) be the any given loss (e.g., train or test error) of f θ ( · ) .

Original: Let E α ( θ 1 , θ 2 ) =
Revised: Let E α ( θ 1 , θ 2 ) =

Original: L ( αθ 1 + (1 − α ) θ 2 ) , for α ∈ [0 , 1] be the loss of the network created by linearly interpolating between parameters of two networks f θ 1 ( · ) and f θ 2 ( · ) .
Revised: L ( αθ 1 + (1 − α ) θ 2 ) , for α ∈ [0 , 1] be the loss of the network created by linearly interpolating between parameters of two networks f θ 1 ( · ) and f θ 2 ( · ) .

Original: The loss barrier B ( θ 1 , θ 2 ) along the linear path between θ 1 and θ 2 is deﬁned as the highest difference between the loss occurred when linearly connecting two points θ 1 , θ 2 and linear interpolation of the loss values at each of them:
Revised: The loss barrier B ( θ 1 , θ 2 ) along the linear path between θ 1 and θ 2 is deﬁned as the highest difference between the loss occurred when linearly connecting two points θ 1 , θ 2 and linear interpolation of the loss values at each of them:

Original: [Equation]
Revised: [Equation]

Original: The above deﬁnition differs from what was proposed by Frankle et al.
Revised: The above deﬁnition differs from what was proposed by Frankle et al.

Original: (2020) in that they used 0 .
Revised: (2020) in that they used 0 .

Original: 5 L ( θ 1 ) + 0 .
Revised: 5 L ( θ 1 ) + 0 .

Original: 5 L ( θ 2 ) instead of α L ( θ 1 )
Revised: 5 L ( θ 2 ) instead of α L ( θ 1 )

Original: + (1 − α
Revised: + (1 − α

Original: ) L ( θ 2 ) in our deﬁnition.
Revised: ) L ( θ 2 ) in our deﬁnition.

Original: These deﬁnitions are the same if L ( θ 1 ) =
Revised: These deﬁnitions are the same if L ( θ 1 ) =

Original: L ( θ 2 ) .
Revised: L ( θ 2 ) .

Original: But if L ( θ 1 ) , L ( θ 2 ) are different, we ﬁnd our deﬁnition to be more appropriate because it assigns no barrier value to a loss that is changing linearly between θ 1 and θ 2 .
Revised: But if L ( θ 1 ) , L ( θ 2 ) are different, we ﬁnd our deﬁnition to be more appropriate because it assigns no barrier value to a loss that is changing linearly between θ 1 and θ 2 .

Original: We say that two networks θ 1 and θ 2 are linear mode connected if the barrier between them along a linear path is ≈ 

Revised: We say that two networks θ 1 and θ 2 are linear mode connected if the barrier between them along a linear path is ≈

Original: (Frankle et al., 2020).
Revised: (Frankle et al., 2020).

Original: It has been observed in the literature that any two minimizers of a deep network can be connected via a non-linear low-loss path (Garipov et al., 2018;
Revised: It has been observed in the literature that any two minimizers of a deep network can be connected via a non-linear low-loss path (Garipov et al., 2018;

Original: Draxler et al., 2018; Fort & Jastrzebski, 2019).
Revised: Draxler et al., 2018; Fort & Jastrzebski, 2019).

Original: This work examines linear mode connectivity (LMC) between minima.
Revised: This work examines linear mode connectivity (LMC) between minima.

Original: Next, we empirically investigate the effect of task difﬁculty and choices such as architecture family, width and depth on LMC of SGD solutions.
Revised: Next, we empirically investigate the effect of task difﬁculty and choices such as architecture family, width and depth on LMC of SGD solutions.

Original: 2. E MPIRICAL I NVESTIGATION : B ARRIERS
Revised: 2. E MPIRICAL I NVESTIGATION : B ARRIERS

Original: In this section, we look into barriers between different SGD solutions on all combinations of four architecture families (MLP (Rosenblatt, 1961), Shallow CNN (Neyshabur, 2020), ResNet (He et al., 2015) and VGG (Simonyan & Zisserman, 2015)) and four datasets (MNIST (LeCun & Cortes, 2010),
Revised: In this section, we look into barriers between different SGD solutions on all combinations of four architecture families (MLP (Rosenblatt, 1961), Shallow CNN (Neyshabur, 2020), ResNet (He et al., 2015) and VGG (Simonyan & Zisserman, 2015)) and four datasets (MNIST (LeCun & Cortes, 2010),

Original: SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 (Krizhevsky et al., 2009)).
Revised: SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 (Krizhevsky et al., 2009)).

Original: The main motivation to use Shallow CNN is to move from fully connected layers (MLP) to convolutions.
Revised: The main motivation to use Shallow CNN is to move from fully connected layers (MLP) to convolutions.

Original: The main difference between Shallow CNN and VGG16 is depth and the main difference between ResNet18 and VGG16 is existence of residual connections.
Revised: The main difference between Shallow CNN and VGG16 is depth and the main difference between ResNet18 and VGG16 is existence of residual connections.

Original: We empirically investigate how different factors such as architecture family, width, depth and task difﬁculty impact the barrier size 2 .
Revised: We empirically investigate how different factors such as architecture family, width, depth and task difﬁculty impact the barrier size 2 .

Original: We refer to training loss barrier as barrier.
Revised: We refer to training loss barrier as barrier.

Original: For loss barriers on a test set see C.3.
Revised: For loss barriers on a test set see E.4.

Original: For train and test errors see A.2 .
Revised: For train and test errors see A.2 .

Original: Width:
Revised: Width:

Original: We evaluate the impact of width on the barrier size in Figure 2.
Revised: We evaluate the impact of width on the barrier size in Figure 2.

Original: We note that for large values of width the barrier becomes small.
Revised: We note that for large values of width the barrier becomes small.

Original: This effect starts at lower width for simpler datasets such as MNIST and SVHN compared to CIFAR datasets.
Revised: This effect starts at lower width for simpler datasets such as MNIST and SVHN compared to CIFAR datasets.

Original: A closer look reveals that the barrier increases with width up to a point and beyond that increasing width leads to lower barrier size.
Revised: A closer look reveals that the barrier increases with width up to a point and beyond that increasing width leads to lower barrier size.

Original: This effect is reminiscent of the double descent phenomena (Belkin et al., 2019; Nakkiran et al., 2019).[Figure] [Figure] VGG
Revised: This effect is reminiscent of the double descent phenomena (Belkin et al., 2019; Nakkiran et al., 2019).

Original: MNISTSVHNCIFAR10CIFAR   50 Depth 0. ResNet MNISTSVHNCIFAR10CIFAR Checking the test error (Figure 8) indicates that in our experiments the barrier peak happens at the same size that needed to ﬁt the training data.
Revised: Checking the test error (Figure 8) indicates that in our experiments the barrier peak happens at the same size that needed to ﬁt the training data.

Original: This phenomena is observed for both fully-connected and convolutional architectures.
Revised: This phenomena is observed for both fully-connected and convolutional architectures.

Original: MLP architectures hit their peak at a lower width compared to CNNs  and a decreasing trend starts earlier.
Revised: MLP architectures hit their peak at a lower width compared to CNNs [Figure] [Figure] MNISTSVHNCIFAR10CIFAR and a decreasing trend starts earlier.

Original: For ResNets the barrier size is saturated at a high value and does not change.
Revised: For ResNets the barrier size is saturated at a high value and does not change.

Original: The barrier value for VGG architecture on different datasets is also saturated at a high value and does not change by increasing the width.
Revised: The barrier value for VGG architecture on different datasets is also saturated at a high value and does not change by increasing the width.

Original: Such similar behavior observed for both ResNets and VGG architectures is due to the effect of depth as discussed in the next paragraph.
Revised: Such similar behavior observed for both ResNets and VGG architectures is due to the effect of depth as discussed in the next paragraph.

Original: Depth: We vary network depth in Figure 3 to evaluate its impact on the barrier between optimal solutions obtained from different initializations.
Revised: Depth: We vary network depth in Figure 3 to evaluate its impact on the barrier between optimal solutions obtained from different initializations.

Original: For MLPs, we ﬁx the layer width at 2 10 while adding identical layers as shown along the x-axis.
Revised: For MLPs, we ﬁx the layer width at 2 10 while adding identical layers as shown along the x-axis.

Original: We observe a fast and signiﬁcant barrier increase as more layers are added.
Revised: We observe a fast and signiﬁcant barrier increase as more layers are added.

Original: For VGG architecture family we observe signiﬁcant barriers.
Revised: For VGG architecture family we observe signiﬁcant barriers.

Original: This might be due to the effect of convolution or depth.
Revised: This might be due to the effect of convolution or depth.

Original: In order to shed light on this observation, we use Shallow
Revised: In order to shed light on this observation, we use Shallow

Original: CNN (Neyshabur, 2020) with only two convolutional layers.
Revised: CNN (Neyshabur, 2020) with only two convolutional layers.

Original: As can be seen in Figure 3 when
Revised: As can be seen in Figure 3 when

Original: Shallow CNN has two layers the barrier size is low, while keeping the layer width ﬁxed at 2 10 and adding more layers increases the barrier size.
Revised: Shallow CNN has two layers the barrier size is low, while keeping the layer width ﬁxed at 2 10 and adding more layers increases the barrier size.

Original: For residual networks we also consider three ResNet architectures with 18, 34 and 50 layers and observe the same barrier sizes as VGG for all these depth values.
Revised: For residual networks we also consider three ResNet architectures with 18, 34 and 50 layers and observe the same barrier sizes as VGG for all these depth values.

Original: The main overall observation from depth experiments is that for both fully-connected and convolutional architectures, increasing depth increases the barrier size signiﬁcantly so the effect of depth is not similar to width.
Revised: The main overall observation from depth experiments is that for both fully-connected and convolutional architectures, increasing depth increases the barrier size signiﬁcantly so the effect of depth is not similar to width.

Original: This can also be attributed to the observations that deeper networks usually have a less smooth landscape (Li et al., 2017).
Revised: This can also be attributed to the observations that deeper networks usually have a less smooth landscape (Li et al., 2017).

Original: Task difﬁculty and architecture choice: In Figure 4 we look into the impact of the task difﬁculty provided by the dataset choice (MNIST, SVHN, CIFAR-10, CIFAR-100, and ImageNet (Deng et al., 2009)) and the architecture type (one-layer MLP with 2 10 neurons, Shallow CNN with two convolutional layer and width of 2 10 , VGG-16 with batch-normalization, ResNet18 and ResNet50).
Revised: Task difﬁculty and architecture choice: In Figure 4 we look into the impact of the task difﬁculty provided by the dataset choice (MNIST, SVHN, CIFAR-10, CIFAR-100, and ImageNet (Deng et al., 2009)) and the architecture type (one-layer MLP with 2 10 neurons, Shallow CNN with two convolutional layer and width of 2 10 , VGG-16 with batch-normalization, ResNet18 and ResNet50).

Original: Each row in Figure 4a and Figure 4b shows the effect of task difﬁculty, e.g. , ﬁxing the task to SVHN and moving from MLP to Shallow CNN gives lower test error hence lower barrier size.
Revised: Each row in Figure 4a and Figure 4b shows the effect of task difﬁculty, e.g. , ﬁxing the task to SVHN and moving from MLP to Shallow CNN gives lower test error hence lower barrier size.

Original: Each column also represents the effect of architecture on a speciﬁc dataset, e.g. , ﬁxing the architecture to Shallow
Revised: Each column also represents the effect of architecture on a speciﬁc dataset, e.g. , ﬁxing the architecture to Shallow

Original: CNN and moving from CIFAR10 to CIFAR100 presents an increase in test error, hence increase in the barrier size.
Revised: CNN and moving from CIFAR10 to CIFAR100 presents an increase in test error, hence increase in the barrier size.

Original: Although deep architectures like VGG16 and ResNet18 present low test error, the discussed effect of depth saturates their barrier at a high level.
Revised: Although deep architectures like VGG16 and ResNet18 present low test error, the discussed effect of depth saturates their barrier at a high level.

Original: Figure 4c aggregates the correlation between test error and size of the barrier.
Revised: Figure 4c aggregates the correlation between test error and size of the barrier.

Original: For MLP and Shallow CNN we observe a high positive 
Revised: For MLP and Shallow CNN we observe a high positive correlation between test error and barrier size across different datasets. Deeper networks (VGGs, ResNets) form a cluster in the top-left, with low test error and high barrier size.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: lower barrier if the test error is lower.
Revised: lower barrier if the test error is lower.

Original: Therefore, any changes in the architecture or the task that improves the test error, also improves the loss barrier.
Revised: Therefore, any changes in the architecture or the task that improves the test error, also improves the loss barrier.

Original: Effect of depth is stronger than (architecture, task) which leads to high barrier values for ResNets on MNIST, SVHN,
Revised: Effect of depth is stronger than (architecture, task) which leads to high barrier values for ResNets on MNIST, SVHN,

Original: CIFAR10, CIFAR100, and ImageNet.
Revised: CIFAR10, CIFAR100, and ImageNet.

Original: correlation between test error and barrier size across different datasets.

Original: Deeper networks (VGGs, ResNets) form a cluster in the top-left, with low test error and high barrier size.

Original:  R OLE OF I NVARIANCE IN L OSS B ARRIERS
Revised:  R OLE OF I NVARIANCE IN L OSS B ARRIERS

Original: Understanding the loss landscape of deep networks has proven to be very challenging.
Revised: Understanding the loss landscape of deep networks has proven to be very challenging.

Original: One of the main challenges in studying the loss landscape without taking the optimization algorithm into account is that there exist many minima with different generalization properties.
Revised: One of the main challenges in studying the loss landscape without taking the optimization algorithm into account is that there exist many minima with different generalization properties.

Original: Most of such minima are not reachable by SGD and we only know about their existence through artiﬁcially-made optimization algorithms and training regimes (Neyshabur et al., 2017).
Revised: Most of such minima are not reachable by SGD and we only know about their existence through artiﬁcially-made optimization algorithms and training regimes (Neyshabur et al., 2017).

Original: To circumvent this issue, we focus on parts of the landscape that are reachable by SGD.
Revised: To circumvent this issue, we focus on parts of the landscape that are reachable by SGD.

Original: Given a dataset and an architecture, one could deﬁne a probability distribution over all solutions reachable by SGD and focus on the subset where SGD is more likely to converge to.
Revised: Given a dataset and an architecture, one could deﬁne a probability distribution over all solutions reachable by SGD and focus on the subset where SGD is more likely to converge to.

Original: 3. I NVARIANCES IN NEURAL NETWORK FUNCTION CLASS
Revised: 3. I NVARIANCES IN NEURAL NETWORK FUNCTION CLASS

Original: We say that a network is invariant with respect to a transformation if and only if the network resulting from the transformation represents the same function as the original network.
Revised: We say that a network is invariant with respect to a transformation if and only if the network resulting from the transformation represents the same function as the original network.

Original: There are two well-known invariances: one is the unit-rescaling due to positive homogeneity of ReLU activations (Neyshabur et al., 2015) and the other is permutation of hidden units.
Revised: There are two well-known invariances: one is the unit-rescaling due to positive homogeneity of ReLU activations (Neyshabur et al., 2015) and the other is permutation of hidden units.

Original: Unit-rescaling has been well-studied and empirical evidence suggests that implicit bias of SGD would make the solution converge to a stage where the weights are more balanced (Neyshabur et al., 2015; Wu et al., 2019).
Revised: Unit-rescaling has been well-studied and empirical evidence suggests that implicit bias of SGD would make the solution converge to a stage where the weights are more balanced (Neyshabur et al., 2015; Wu et al., 2019).

Original: Since we are interested in the loss landscape through the lens of SGD and SGD is much more likely to converge to a particular rescaling, consideration of this type of invariance does not seem useful.
Revised: Since we are interested in the loss landscape through the lens of SGD and SGD is much more likely to converge to a particular rescaling, consideration of this type of invariance does not seem useful.

Original: However, in the case of permutations, all permutations are equally likely for SGD and therefore, it is important to understand their role in the geometric properties of the landscape and its basins of attraction.
Revised: However, in the case of permutations, all permutations are equally likely for SGD and therefore, it is important to understand their role in the geometric properties of the landscape and its basins of attraction.

Original: Here we consider invariances that are in form of permutations of hidden units in each layer of the network, i.e. , each layer i with parameters
Revised: Here we consider invariances that are in form of permutations of hidden units in each layer of the network, i.e. , each layer i with parameters

Original: W i is replaced with P
Revised: W i is replaced with P

Original: i W i P i − 1 where P i is a permutation matrix and P l = P 0 is the identity matrix.
Revised: i W i P i − 1 where P i is a permutation matrix and P l = P 0 is the identity matrix.

Original: Note that our results only hold for permutation matrices since only permutation commutes with nonlinearity.
Revised: Note that our results only hold for permutation matrices since only permutation commutes with nonlinearity.

Original: We use P to refer to the set of valid permutations for a neural network and use π to refer to a valid permutation.
Revised: We use P to refer to the set of valid permutations for a neural network and use π to refer to a valid permutation.

Original: 3. O UR C ONJECTURE
Revised: 3. O UR C ONJECTURE

Original: As mentioned above, SGD’s implicit regularization balances weight norms and, therefore, scale invariance does not seem to play an important role in understanding symmetries of solutions found by SGD.
Revised: As mentioned above, SGD’s implicit regularization balances weight norms and, therefore, scale invariance does not seem to play an important role in understanding symmetries of solutions found by SGD.

Original: Consequently, here we focus on permutation invariance and conjecture that taking it into account allows us to have a much simpler view of SGD solutions.
Revised: Consequently, here we focus on permutation invariance and conjecture that taking it into account allows us to have a much simpler view of SGD solutions.

Original: We ﬁrst state our conjecture informally:
Revised: We ﬁrst state our conjecture informally:

Original: Most SGD solutions belong to a set S whose elements can be permuted in such a way that there is no barrier on the linear interpolation between any two permuted elements in S .
Revised: Most SGD solutions belong to a set S whose elements can be permuted in such a way that there is no barrier on the linear interpolation between any two permuted elements in S .

Original: The above conjecture suggests that most SGD solutions end up in the same basin in the loss landscape after proper permutation (see Figure 1 left panel).
Revised: The above conjecture suggests that most SGD solutions end up in the same basin in the loss landscape after proper permutation (see Figure 1 left panel).

Original: We acknowledge that the above conjecture is bold.
Revised: We acknowledge that the above conjecture is bold.

Original: Nonetheless, we argue that coming up with strong conjectures and attempting to disprove them is an effective method for scientiﬁc progress.
Revised: Nonetheless, we argue that coming up with strong conjectures and attempting to disprove them is an effective method for scientiﬁc progress.

Original: Note, our conjecture also has great practical implications for model ensembling and parallelism since one can average models that are in the same basin in the loss landscape.
Revised: Note, our conjecture also has great practical implications for model ensembling and parallelism since one can average models that are in the same basin in the loss landscape.

Original: The conjecture can be formalized as follows:
Revised: The conjecture can be formalized as follows:

Original: Conjecture 1.
Revised: Conjecture 1.

Original: Let f ( θ ) be the function representing a feedforward network with parameters θ ∈ R k ,
Revised: Let f ( θ ) be the function representing a feedforward network with parameters θ ∈ R k ,

Original: P be the set of all valid permutations for the network, P : R k × P → R k be the function that applies a given permutation to parameters and returns the permuted version, and B ( · , · ) be the function that returns barrier value between two solutions as deﬁned in equation 1.
Revised: P be the set of all valid permutations for the network, P : R k × P → R k be the function that applies a given permutation to parameters and returns the permuted version, and B ( · , · ) be the function that returns barrier value between two solutions as deﬁned in Equation 1.

Original: Then, there exists a width h >such that for any network f ( θ ) of width at least h the following holds: There exist a set of solutions S ⊆ R k and a function Q : S → P such for any θ 1 , θ 2 ∈ S , B ( P ( θ 1 , Q ( θ 1 )) , θ 2 ) ≈ 0 and with high probability over an SGD solution θ , we have θ ∈ S .
Revised: Then, there exists a width h > 0 such that for any network f ( θ ) of width at least h the following holds: There exist a set of solutions S ⊆ R k and a function Q : S → P such for any θ 1 , θ 2 ∈ S , B ( P ( θ 1 , Q ( θ 1 )) , θ 2 ) ≈ 0 and with high probability over an SGD solution θ , we have θ ∈ S .

Original: Next, we approach Conjecture 1 from both theoretical and empirical aspects and provide some evidence to support it.
Revised: Next, we approach Conjecture 1 from both theoretical and empirical aspects and provide some evidence to support it.

Original: 3. A THEORETICAL RESULT
Revised: 3. A THEORETICAL RESULT

Original: In this section we provide elementary theoretical results in support of our conjecture.
Revised: In this section we provide elementary theoretical results in support of our conjecture.

Original: Although the theoretical result is provided for a very limited setting, we believe it helps us understand the mechanism that could give rise to our conjecture.
Revised: Although the theoretical result is provided for a very limited setting, we believe it helps us understand the mechanism that could give rise to our conjecture.

Original: Bellow, we theoretically show that
Revised: Bellow, we theoretically show that

Original: Conjectureholds for a fully-connected network with a single hidden layer at initialization.
Revised: Conjectureholds for a fully-connected network with a single hidden layer at initialization.

Original: Proof is given in
Revised: Proof is given in

Original: Appendix B.
Revised: Appendix D.

Original: Theorem 1.
Revised: Theorem 3.1.

Original: Let f v , U ( x ) = v (cid:62) σ ( Ux ) be a fully-connected network with h hidden units where σ ( · ) is ReLU activation, v ∈ R h and U ∈ R
Revised: Let f v , U ( x ) = v (cid:62) σ ( Ux ) be a fully-connected network with h hidden units where σ ( · ) is ReLU activation, v ∈ R h and U ∈ R

Original: h × d are the parameters and x ∈ R
Revised: h × d are the parameters and x ∈ R

Original: d is the input.
Revised: d is the input.

Original: If each element of U and U (cid:48) is sampled uniformly from [ − 1 / √ d, 1 / √ d ] and each element of v and v (cid:48) is sampled uniformly from [ − 1 / √ h, 1 / √ h ] , then for any x ∈ R d such that (cid:107) x (cid:107) 2 = √ d , with probability 1 − δ over U , U (cid:48) , v , v (cid:48) , there exist a permutation such that
Revised: If each element of U and U (cid:48) is sampled uniformly from [ − 1 / √ d, 1 / √ d ] and each element of v and v (cid:48) is sampled uniformly from [ − 1 / √ h, 1 / √ h ] , then for any x ∈ R d such that (cid:107) x (cid:107) 2 = √ d , with probability 1 − δ over U , U (cid:48) , v , v (cid:48) , there exist a permutation such that

Original: [Equation]
Revised: [Equation]

Original: where v (cid:48)(cid:48) and U (cid:48)(cid:48) are permuted versions of v (cid:48) and U (cid:48) .
Revised: where v (cid:48)(cid:48) and U (cid:48)(cid:48) are permuted versions of v (cid:48) and U (cid:48) .

Original: Theorem 1 states that for wide enough fully-connected networks with a single hidden layer, one can ﬁnd a permutation that leads to having no barrier at random initialization.
Revised: Theorem 3.1 states that for wide enough fully-connected networks with a single hidden layer, one can ﬁnd a permutation that leads to having no barrier at random initialization.

Original: Although, our prove only covers random initialization, we believe with a more involved proof, it might be possible to extend it to NTK regime (Jacot et al., 2018).
Revised: Although, our prove only covers random initialization, we believe with a more involved proof, it might be possible to extend it to NTK regime (Jacot et al., 2018).

Original: We leave this for future work.
Revised: We leave this for future work.

Original: 3. D IRECT EMPIRICAL EVALUATION OF C ONJECTURE
Revised: 3. D IRECT EMPIRICAL EVALUATION OF C ONJECTURE

Original: Another possible approach is to use brute-force (BF) search mechanism and ﬁnd the function Q for elements of S .
Revised: Another possible approach is to use brute-force (BF) search mechanism and ﬁnd the function Q for elements of S .

Original: The factorial growth of the number of permutations with the size of hidden units in each layer hinders exhaustive search for a winning permutation π to linear mode connect
Revised: The factorial growth of the number of permutations with the size of hidden units in each layer hinders exhaustive search for a winning permutation π to linear mode connect

Original: P ( θ 1 , π ) and θ 2 .
Revised: P ( θ 1 , π ) and θ 2 .

Original: Even for MLPs with just one hidden layer brute-force works in reasonable time up to 2 4 neurons only, forcing the search to examine 2 4 !
Revised: Even for MLPs with just one hidden layer brute-force works in reasonable time up to 2 4 neurons only, forcing the search to examine 2 4 !

Original: ≈ 2 · 10 13 permuted networks.
Revised: ≈ 2 · 10 13 permuted networks.

Original: BF is not feasible even for modest size deep networks.
Revised: BF is not feasible even for modest size deep networks.

Original: For small networks, one can use BF to ﬁnd permutations between different models (see C.2).
Revised: For small networks, one can use BF to ﬁnd permutations between different models (see E.3).

Original: However, small size networks are not the focus of this paper and
Revised: However, small size networks are not the focus of this paper and

Original: Conjecture 1 speciﬁcally mentions that.
Revised: Conjecture 1 speciﬁcally mentions that.

Original: Given the size of search space, using a more advanced search algorithm can be useful.
Revised: Given the size of search space, using a more advanced search algorithm can be useful.

Original: The issue with this approach is that since it relies on the strength of a search algorithm, if the search algorithm fails in ﬁnding the permutation, one cannot be sure about the source of failure being the search algorithm or nonexistence of a permutation that leads to no barrier. 
Revised: The issue with this approach is that since it relies on the strength of a search algorithm, if the search algorithm fails in ﬁnding the permutation, one cannot be sure about the source of failure being the search algorithm or nonexistence of a permutation that leads to no barrier. 3. O UR MODEL VS REAL WORLD : A N ALTERNATIVE APPROACH We propose the following approach to circumvent the above obstacles. We create a competing set S (cid:48)

Revised: (our model) as a proxy for set S (real world).

Revised: Given an SGD solution θ 1 ∈ S , we deﬁne

Original: 2 3 2 5 2 7 2 9 2 11 2 13 2 15 Width 0.
Revised: 2 3 2 5 2 7 2 9 2 11 2 13 2 15 Width 0.

Original: B a rr i e r
Revised: B a rr i e r

Original: MLP 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 Width 0.
Revised: MLP 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 Width 0.

Original: Shallow CNN 1 8 Depth 0.
Revised: Shallow CNN 1 8 Depth 0.

Original: MLP   8 Depth 0.
Revised: MLP   8 Depth 0.

Original: Shallow-CNN
Revised: Shallow-CNN

Original: MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR
Revised: MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR

Original: [Figure]
Revised: [Figure]

Original: 3. O UR MODEL VS REAL WORLD : A N ALTERNATIVE APPROACH

Original: We propose the following approach to circumvent the above obstacles.

Original: We create a competing set S (cid:48)

Original: (our model) as a proxy for set S (real world).

Original: Given an SGD solution θ 1 ∈ S , we deﬁne

Original: S (cid:48)
Revised: S (cid:48)

Original: = { P ( θ 1 , π ) |∀ π ∈ P} .
Revised: = { P ( θ 1 , π ) |∀ π ∈ P} .

Original: We know that set S (cid:48) satisﬁes the conjecture.
Revised: We know that set S (cid:48) satisﬁes the conjecture.

Original: For set S (cid:48) , all points are known permutations of θ 1 .
Revised: For set S (cid:48) , all points are known permutations of θ 1 .

Original: Therefore, one can permute all points to remove their barriers with θ 1 , i.e, for all θ 2 = P ( θ 1 , π ) , one can use Q ( θ 2 ) =
Revised: Therefore, one can permute all points to remove their barriers with θ 1 , i.e, for all θ 2 = P ( θ 1 , π ) , one can use Q ( θ 2 ) =

Original: π − 1 to remove the barrier between θ 1 and θ 2 .
Revised: π − 1 to remove the barrier between θ 1 and θ 2 .

Original: Our goal is therefore to show that S is similar to S (cid:48) in terms of barrier behavior.
Revised: Our goal is therefore to show that S is similar to S (cid:48) in terms of barrier behavior.

Original: Equivalence of S (cid:48) and S in terms of barriers, means that if we choose an element θ in S , one should be able to ﬁnd permutations for each of other elements of the S so that the permuted elements have no barrier with θ and hence are in the same basin as θ .
Revised: Equivalence of S (cid:48) and S in terms of barriers, means that if we choose an element θ in S , one should be able to ﬁnd permutations for each of other elements of the S so that the permuted elements have no barrier with θ and hence are in the same basin as θ .

Original: The consequence of the equivalence of our model to real world is that Conjecture 1 holds.
Revised: The consequence of the equivalence of our model to real world is that Conjecture 1 holds.

Original: The conjecture effectively means that different basins exist because of the permutation invariance and if permutation invariance is taken into account (by permuting solutions to remove the barriers between them), there is only one basin, i.e., all solutions reside in the same basin in the loss landscape.
Revised: The conjecture effectively means that different basins exist because of the permutation invariance and if permutation invariance is taken into account (by permuting solutions to remove the barriers between them), there is only one basin, i.e., all solutions reside in the same basin in the loss landscape.

Original: We actually want to show S is similar to S (cid:48) in terms of optimizing over all permutations but that is not possible so we show S is similar to S (cid:48) in terms of barrier without search or when we search over a smaller set of permutations using a search algorithm.
Revised: We actually want to show S is similar to S (cid:48) in terms of optimizing over all permutations but that is not possible so we show S is similar to S (cid:48) in terms of barrier without search or when we search over a smaller set of permutations using a search algorithm.

Original: In the next section, we investigate our conjecture using this approach.
Revised: In the next section, we investigate our conjecture using this approach.

Original:  E MPIRICAL I NVESTIGATION
Revised:  E MPIRICAL I NVESTIGATION

Original: In this section we show that S and S (cid:48) have similar loss barrier along different factors such as width, depth, architecture, dataset and other model parameters (with and without searching for a permutation that reduces the barrier), hence supporting our conjecture.
Revised: In this section we show that S and S (cid:48) have similar loss barrier along different factors such as width, depth, architecture, dataset and other model parameters (with and without searching for a permutation that reduces the barrier), hence supporting our conjecture.

Original: As discussed in Section 2, the barrier for both VGG and ResNet architectures is saturated at a high value hinting that the loss landscape might be more complex for these architecture families.
Revised: As discussed in Section 2, the barrier for both VGG and ResNet architectures is saturated at a high value hinting that the loss landscape might be more complex for these architecture families.

Original: In our experiments we observed that S and S (cid:48) have similar high barriers for both of these architectures (see Appendix C.1).
Revised: In our experiments we observed that S and S (cid:48) have similar high barriers for both of these architectures (see Appendix E.2).

Original: Moreover, we observed that for both S and S (cid:48) the employed algorithms (Section 4.2) were unable to ﬁnd a permutation to reduce the barrier and hence our model shows a similar behavior to real world 3 .
Revised: Moreover, we observed that for both S and S (cid:48) the employed algorithms (Section 4.2) were unable to ﬁnd a permutation to reduce the barrier and hence our model shows a similar behavior to real world 3 .

Original: Given that the width and depth do not inﬂuence the barrier behavior in VGG and ResNet architectures, here we only focus on the effect of width and depth on barrier sizes for MLPs and Shallow CNNs.
Revised: Given that the width and depth do not inﬂuence the barrier behavior in VGG and ResNet architectures, here we only focus on the effect of width and depth on barrier sizes for MLPs and Shallow CNNs.

Original: 4. S IMILARITY OF S AND S (cid:48)
Revised: 4. S IMILARITY OF S AND S (cid:48)

Original: Figure 5 compares our model to the real world and shows that S (cid:48) and S have strikingly similar barriers as we change different architecture parameters such as width and depth across various architecture families and datasets.
Revised: Figure 5 compares our model to the real world and shows that S (cid:48) and S have strikingly similar barriers as we change different architecture parameters such as width and depth across various architecture families and datasets.

Original: This surprising level of similarity between our model and real world on variety of settings provide strong evidence for the conjecture.
Revised: This surprising level of similarity between our model and real world on variety of settings provide strong evidence for the conjecture.

Original: Even if the conjecture is not precisely correct as stated, the empirical results suggest that the structural similarities between our model and real world makes our model a useful simpliﬁcation of the real world for studying the loss landscape.
Revised: Even if the conjecture is not precisely correct as stated, the empirical results suggest that the structural similarities between our model and real world makes our model a useful simpliﬁcation of the real world for studying the loss landscape.

Original: For example, the effect of width and depth on the barrier is almost identical in our model and the real world which suggests that permutations are perhaps playing the main role in such behaviors.
Revised: For example, the effect of width and depth on the barrier is almost identical in our model and the real world which suggests that permutations are perhaps playing the main role in such behaviors.

Revised: 4. S EARCH ALGORITHMS FOR FINDING A WINNING PERMUTATION

Revised: The problem of ﬁnding a winning permutation π ∈ P is a variant of the Travelling Salesman Problem where neurons are mapped to cities visited by a salesman.

Revised: The problem belongs to the class of

Revised: NP-hard optimization problems and simulated annealing (SA) is often used to ﬁnd a solution for

Original: 1: procedure SA( { θ i } , i = 1 ..n, n ≥ 2 ) (cid:46) Goal: minimize the barrier between n solutions 2: π i = π 0 , ∀ i = 1 ..n 3: for k = 0 ; k < k max ; k
Revised: 1: procedure SA( { θ i } , i = 1 ..n, n ≥ 2 ) (cid:46) Goal: minimize the barrier between n solutions 2: π i = π 0 , ∀ i = 1 ..n 3: for k = 0 ; k < k max ; k

Original: ++ do 4:
Revised: ++ do 4:

Original: T ← temperature ( k +1 k max ) 5:
Revised: T ← temperature ( k +1 k max ) 5:

Original: Pick random candidate permutations { ˆ π i } , ∀ i = 1 ..n 6: if Ψ( P ( θ i , ˆ π i )) <
Revised: Pick random candidate permutations { ˆ π i } , ∀ i = 1 ..n 6: if Ψ( P ( θ i , ˆ π i )) <

Original: Ψ( P ( θ i , π i )) then (cid:46) Ψ : barrier objective function 7: π i ← ˆ π i return { π i }
Revised: Ψ( P ( θ i , π i )) then (cid:46) Ψ : barrier objective function 7: π i ← ˆ π i return { π i }

Original: [Figure]
Revised: [Figure]

Original: The problem of ﬁnding a winning permutation π ∈ P is a variant of the Travelling Salesman Problem where neurons are mapped to cities visited by a salesman. The problem belongs to the class of NP-hard optimization problems and simulated annealing (SA) is often used to ﬁnd a solution for such a combinatorial search problem.
Revised: such a combinatorial search problem.

Original: SA’s performance however highly depends on the parameter choices, including the minimum and maximum temperatures, the cooling schedule and the number of optimization steps.
Revised: SA’s performance however highly depends on the parameter choices, including the minimum and maximum temperatures, the cooling schedule and the number of optimization steps.

Original: The pseudocode of SA is shown in Algorithm 1.
Revised: The pseudocode of SA is shown in Algorithm 1.

Original: SA takes a set of solutions { θ i } , i = 1 ..n, n ≥ 2 as input (we use n = 5 ) and searches for a set of permutations { π i } that reduce the barriers between all permuted (cid:0) n 2 (cid:1) solution pairs.
Revised: SA takes a set of solutions { θ i } , i = 1 ..n, n ≥ 2 as input (we use n = 5 ) and searches for a set of permutations { π i } that reduce the barriers between all permuted (cid:0) n 2 (cid:1) solution pairs.

Original: To ﬁnd the best { π i } , in each step of SA the current candidate permutations { ˆ π i } , i = 1 ..n are evaluated to minimize the objective function Ψ .
Revised: To ﬁnd the best { π i } , in each step of SA the current candidate permutations { ˆ π i } , i = 1 ..n are evaluated to minimize the objective function Ψ .

Original: We use two versions of simulated annealing that vary in their deﬁnition of Ψ to evaluate the conjecture.
Revised: We use two versions of simulated annealing that vary in their deﬁnition of Ψ to evaluate the conjecture.

Original: Simulated Annealing 1 (SA 1 ) .
Revised: Simulated Annealing 1 (SA 1 ) .

Original: In the ﬁrst version SA 1 , Ψ is deﬁned as the average pairwise barrier between candidate permutations B ( P ( θ i , π i ) , P ( θ j , π j )) , i (cid:54) = j .
Revised: In the ﬁrst version SA 1 , Ψ is deﬁned as the average pairwise barrier between candidate permutations B ( P ( θ i , π i ) , P ( θ j , π j )) , i (cid:54) = j .

Original: Simulated Annealing 2 (SA 2 ) .
Revised: Simulated Annealing 2 (SA 2 ) .

Original: In the second version SA 2 , we average the weights of permuted models P ( θ i , π i ) ﬁrst and deﬁned Ψ as the train error of the resulting average model.
Revised: In the second version SA 2 , we average the weights of permuted models P ( θ i , π i ) ﬁrst and deﬁned Ψ as the train error of the resulting average model.

Original: The simplest form of SA 2 happens if n = 2 and is discussed in Section A.3.
Revised: The simplest form of SA 2 happens if n = 2 and is discussed in Section A.3.

Original: The rationale behind these two versions is that if the solutions reside in one basin, there is no barrier between them.
Revised: The rationale behind these two versions is that if the solutions reside in one basin, there is no barrier between them.

Original: Therefore averaging solutions in one basin yields another solution inside their convex hull.
Revised: Therefore averaging solutions in one basin yields another solution inside their convex hull.

Original: Although each version of SA has a different deﬁnition of the objective function Ψ to ﬁnd the best permutation, for all SA versions we report the average barrier between all pairs in the plots.
Revised: Although each version of SA has a different deﬁnition of the objective function Ψ to ﬁnd the best permutation, for all SA versions we report the average barrier between all pairs in the plots.

Original: Our empirical results suggest that SA 1 and SA 2 yield very similar performance.
Revised: Our empirical results suggest that SA 1 and SA 2 yield very similar performance.

Original: However, SAis signiﬁcantly less computationally expensive, which makes it more suitable for exploring larger models.
Revised: However, SAis signiﬁcantly less computationally expensive, which makes it more suitable for exploring larger models.

Original: In the following sections we present the results obtained with SA 2 only and refer to this version as SA.
Revised: In the following sections we present the results obtained with SA 2 only and refer to this version as SA.

Original: For more details on SA implementation see Appendix A.4.
Revised: For more details on SA implementation see Appendix A.4.

Original: The left two plots in Figure 6 show that SA 2 is not able to ﬁnd permutations that improve pair-wise barrier signiﬁcantly.
Revised: The left two plots in Figure 6 show that SA 2 is not able to ﬁnd permutations that improve pair-wise barrier signiﬁcantly.

Original: We know that SA does not guarantee ﬁnding a solution and is known to lose its effectiveness on TSP benchmarks beyond 1’000 cities (Zhan et al., 2016).
Revised: We know that SA does not guarantee ﬁnding a solution and is known to lose its effectiveness on TSP benchmarks beyond 1’000 cities (Zhan et al., 2016).

Original: The effectiveness of SA is also reduced here as we can only evaluate the cost of full route (divide and conquer is not possible).
Revised: The effectiveness of SA is also reduced here as we can only evaluate the cost of full route (divide and conquer is not possible).

Original: One way to increase this effectiveness is to reduce the search space which we will discuss next.
Revised: One way to increase this effectiveness is to reduce the search space which we will discuss next.

Original: Search space reduction .
Revised: Search space reduction .

Original: In order to reduce the search space, here we only take two SGD solutions θ 1 and θ 2 , permute θ 1 and report the barrier between permuted θ 

Revised: In order to reduce the search space, here we only take two SGD solutions θ 1 and θ 2 , permute θ 1 and report the barrier between permuted θ

Original: and θ 2 as found by SA with 
Revised: and θ 2 as found by SA with n = 2 .

Revised: The right two plots in Figure 6 shows this intervention helps SA to ﬁnd better permutations.

Revised: In particular, the barrier improves signiﬁcantly for MNIST and SVHN datasets for both MLP and

Revised: Shallow CNN across different width.

Revised: However, similar to Section 2, we did not observe signiﬁcant improvements when increasing depth (see Figure 12).

Original: 2 3 2 5 2 7 2 9 2 11 2 13 2 15 Width 0.
Revised: 2 3 2 5 2 7 2 9 2 11 2 13 2 15 Width 0.

Original: B a rr i e r
Revised: B a rr i e r

Original: MLP 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 Width 0.
Revised: MLP 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 Width 0.

Original: Shallow CNN 1 8 Depth 0.
Revised: Shallow CNN 1 8 Depth 0.

Original: MLP   8 Depth 0.
Revised: MLP   8 Depth 0.

Original: Shallow-CNN
Revised: Shallow-CNN

Original: MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR
Revised: MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR

Original: [Figure]
Revised: [Figure]

Original: n = 2 .

Original: The right two plots in Figure 6 shows this intervention helps SA to ﬁnd better permutations.

Original: In particular, the barrier improves signiﬁcantly for MNIST and SVHN datasets for both MLP and

Original: Shallow CNN across different width.

Original: However, similar to Section 2, we did not observe signiﬁcant improvements when increasing depth (see Figure 12)

Original: 4. S IMILARITY OF S AND S (cid:48) AFTER SEARCH
Revised: S IMILARITY OF S AND S (cid:48) AFTER SEARCH

Original: Figure 7 shows the surprising similarity of the barrier between S and S(cid:48) even after applying a permutation found by a search algorithm (when search space is reduced).
Revised: Figure 7 shows the surprising similarity of the barrier between S and S even after applying a permutation found by a search algorithm (when search space is reduced).

Original: We also observe that reducing the search space makes SA more successful in ﬁnding the permutation { π } to remove the barriers.
Revised: We also observe that reducing the search space makes SA more successful in ﬁnding the permutation { π } to remove the barriers.

Original: Speciﬁcally, in some cases SA could indeed ﬁnd permutations that when applied to θ 1 result in zero barrier.
Revised: Speciﬁcally, in some cases SA could indeed ﬁnd permutations that when applied to θ 1 result in zero barrier.

Original: However, the fact that SA’s success shows a similar pattern for S , S (cid:48) provides another evidence in support of the conjecture.
Revised: However, the fact that SA’s success shows a similar pattern for S , S (cid:48) provides another evidence in support of the conjecture.

Original: For example, SA successfully reduces the barrier for both
Revised: For example, SA successfully reduces the barrier for both

Original: S , S (cid:48) on MNIST and SVHN datasets.
Revised: S , S (cid:48) on MNIST and SVHN datasets.

Original: Figure 1 (right) summarizes our extensive empirical evidence (more than 3000 trained networks) in one density plot, supporting similarity of barriers in real world and our model across different choices of architecture family, dataset, width, depth, and random seed.
Revised: Figure 1 (right) summarizes our extensive empirical evidence (more than 3000 trained networks) in one density plot, supporting similarity of barriers in real world and our model across different choices of architecture family, dataset, width, depth, and random seed.

Original: Putting together, all our empirical results support our main conjecture.
Revised: Putting together, all our empirical results support our main conjecture.

Original:  D ISCUSSIONS AND C ONCLUSION
Revised:  D ISCUSSIONS AND C ONCLUSION

Original: We investigated the loss landscape of ReLU networks and proposed and probed the conjecture that the barriers in the loss landscape between different solutions of a neural network optimization problem are an artifact of ignoring the permutation invariance of the function class.
Revised: We investigated the loss landscape of ReLU networks, proposed and probed the conjecture that the barriers in the loss landscape between different solutions of a neural network optimization problem are an artifact of ignoring the permutation invariance of the function class.

Original: In a nutshell, this conjecture suggests that if one considers permutation invariance, there is essentially no loss barrier between different solutions and they all exist in the same basin in the loss landscape.
Revised: In a nutshell, this conjecture suggests that if one considers permutation invariance, there is essentially no loss barrier between different solutions and they all exist in the same basin in the loss landscape.

Original: Our analysis has direct implication on initialization schemes for neural networks.
Revised: Our analysis has direct implication on initialization schemes for neural networks.

Original: Essentially it postulates that randomness in terms of permutation does not impact the quality of the ﬁnal result.
Revised: Essentially it postulates that randomness in terms of permutation does not impact the quality of the ﬁnal result.

Original: It is interesting to explore whether it is possible to come up with an initialization that does not have permutation invariance and only acts like a perturbation to the same permutation.
Revised: It is interesting to explore whether it is possible to come up with an initialization that does not have permutation invariance and only acts like a perturbation to the same permutation.

Original: If all basins in the loss landscape are basically the same function, there will be no need to search all of them.
Revised: If all basins in the loss landscape are basically the same function, there will be no need to search all of them.

Original: One can explore the same basin while looking for diverse solutions and this makes search much easier and would lead to substantially more efﬁcient search algorithms.
Revised: One can explore the same basin while looking for diverse solutions and this makes search much easier and would lead to substantially more efﬁcient search algorithms.

Original: Another area where our analysis is of importance is for ensembles and distributed training.
Revised: Another area where our analysis is of importance is for ensembles and distributed training.

Revised: Related works (Frankle et al., 2020; Fort et al., 2019) show that simply averaging two SGD solutions would fail.

Revised: If these models lie at the periphery of a wide and ﬂat low loss region then ensembling them in their weight space (averaging), creates a model tending to the center of the region, which leads to performance improvement (Izmailov et al., 2019; Wen et al., 2020).

Original: If we can track the optimal permutation (that brings all solutions to one basin), it is possible to use it to do weight averaging and build ensembles more efﬁciently.
Revised: If we can track the optimal permutation (that brings all solutions to one basin), it is possible to use it to do weight averaging and build ensembles more efﬁciently.

Original: Moreover, we are interested in answering the question whether there is a one-to-one mapping between lottery tickets and permutations. 
Revised: Moreover, we are interested in answering the question whether there is a one-to-one mapping between lottery tickets and permutations. Frankle et al.

Revised: (2020) requires stability to ﬁnd lottery tickets.

Revised: They deﬁne stability as the point in training trajectory where, if we branch at this point and train two copies with different seeds, the trained solutions are linearly mode connected.

Revised: We conjecture that all the SGD trained solutions are linearly mode connected if the permutation is considered (satisfying the necessary condition for Lottery Ticket Hypothesis).

Original: We believe our analysis laid the ground for investigating these important questions and testing the usefulness of our conjecture in ensemble methods and pruning, which is the subject of future studies.
Revised: We believe our analysis laid the ground for investigating these important questions and testing the usefulness of our conjecture in ensemble methods and pruning, which is the subject of future studies.

Original: The biggest limiting factor of our study is the size of the search space and hence we need a strong search algorithm, specially for deep models where the size and complexity of the search space was prohibitive in terms of computation for the existing search methods.
Revised: The biggest limiting factor of our study is the size of the search space and hence we need a strong search algorithm, specially for deep models where the size and complexity of the search space was prohibitive in terms of computation for the existing search methods.

Original: We hope improvements of search algorithms can help us to extend these results.
Revised: We hope improvements of search algorithms can help us to extend these results.

Original: Lastly, our analysis focuses on image recognition task and extending the results to natural language tasks is of interest for future work.
Revised: Lastly, our analysis focuses on image recognition task and extending the results to natural language tasks is of interest for future work.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: All the experiments conducted in this paper are based on open-sourced publicly available code at https://github.com/AnonymousMLSubmission/PermutationInvariance .
Revised: All the experiments conducted in this paper are based on open-sourced publicly available code at https://github.com/AnonymousMLSubmission/PermutationInvariance .

Original: All the implementation details, including training hyper-parameters, Train and Test performance of the trained SGD solutions,
Revised: All the implementation details, including training hyper-parameters, Train and Test performance of the trained SGD solutions,

Original: Simulated Annealing hyper-parameters, etc. are shared in Appendix A. Proof of the theorem is also provided in Appendix B.
Revised: Simulated Annealing hyper-parameters, etc. are shared in Appendix A. Proof of the theorem is also provided in Appendix D

Original: A PPENDIX A I MPLEMENTATION DETAILS
Revised: A PPENDIX A I MPLEMENTATION DETAILS

Original: We used Caliban (Ritchie et al., 2020) to manage all experiments in a reproducible environment in
Revised: We used Caliban (Ritchie et al., 2020) to manage all experiments in a reproducible environment in

Original: Google Cloud’s AI Platform.
Revised: Google Cloud’s AI Platform.

Original: Each point in plots show the mean value taken over 10 different runs (20 trained networks).
Revised: Each point in plots show the mean value taken over 10 different runs (20 trained networks).

Original: A. T RAINING H YPER - PARAMETERS
Revised: A. T RAINING H YPER - PARAMETERS

Original: Table 1 summarizes the set of used hyper-parameters for training different networks.
Revised: Table 1 summarizes the set of used hyper-parameters for training different networks.

Original: [Table]
Revised: [Table]

Original: A. P ERFORMANCE E VALUATION OF T RAINED M ODELS : E RROR AND L OSS
Revised: A. P ERFORMANCE E VALUATION OF T RAINED M ODELS : E RROR AND L OSS

Original: Figure 8 demonstrate Train and Test error/loss for experiments on the effect of width.
Revised: Figure 8 demonstrate Train and Test error/loss for experiments on the effect of width.

Original: Here for MLP we have one hidden layer, for Shallow CNN two convolutional layer, ResNet is ﬁxed to ResNet18, and VGG16 is selected from VGG family.
Revised: Here for MLP we have one hidden layer, for Shallow CNN two convolutional layer, ResNet is ﬁxed to ResNet18, and VGG16 is selected from VGG family.

Original: Figure 9 also demonstrates Train and Test error/loss for different depth.
Revised: Figure 9 also demonstrates Train and Test error/loss for different depth.

Original: We set MLP to have 1024 hidden units in each layer, Shallow CNN, VGG and
Revised: We set MLP to have 1024 hidden units in each layer, Shallow CNN, VGG and

Original: ResNest to have 1024, 64, 64 channels in each convolutional layer, respectively
Revised: ResNest to have 1024, 64, 64 channels in each convolutional layer, respectively

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. S IMILARITY OF S AND S (cid:48) : D ETAILED VIEW
Revised: A. S IMILARITY OF S AND S (cid:48) : D ETAILED VIEW

Original: Aggregated empirical results .
Revised: Aggregated empirical results .

Original: Figure 10 shows the aggregation of our extensive empirical evidence (more than 3000 trained networks) in one plot comparing barriers in real world against our model across different choices of architecture family, dataset, width, depth
Revised: Figure 10 shows the aggregation of our extensive empirical evidence (more than 3000 trained networks) in one plot comparing barriers in real world against our model across different choices of architecture family, dataset, width, depth

Original: [Figure]
Revised: [Figure]

Original: and random seed.
Revised: and random seed.

Original: Points with solid edges correspond to lowest barrier found after searching in the space of valid permutations using a Simulated Annealing (SA).
Revised: Points with solid edges correspond to lowest barrier found after searching in the space of valid permutations using a Simulated Annealing (SA).

Original: SA shows better performance for shallow networks pushing more solid edge points to the lower left corner
Revised: SA shows better performance for shallow networks pushing more solid edge points to the lower left corner

Original: [Figure]
Revised: [Figure]

Original: Simulated Annealing performance .
Revised: Simulated Annealing performance .

Original: In this Section, we show that S and S (cid:48) have similar loss barriers we change different architecture parameters such as as width, depth across various datasets.
Revised: In this Section, we show that S and S (cid:48) have similar loss barriers we change different architecture parameters such as as width, depth across various datasets.

Original: Here we consider the mean over all pair-wise barriers as barrier size.
Revised: Here we consider the mean over all pair-wise barriers as barrier size.

Original: Figure 5 shows the similarity of
Revised: Figure 5 shows the similarity of

Original: S and S (cid:48) before permutation.
Revised: S and S (cid:48) before permutation.

Original: In each plot, barrier size for S is similar to S (cid:48) .
Revised: In each plot, barrier size for S is similar to S (cid:48) .

Original: Such similarity is also observed in Figure 11, where we see the barrier after permutation using SA.
Revised: Such similarity is also observed in Figure 11, where we see the barrier after permutation using SA.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR
Revised: MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR100MNISTSVHNCIFAR10CIFAR

Original: [Figure]
Revised: [Figure]

Original: Search space reduction .
Revised: Search space reduction .

Original: In order to reduce the search space, here we only take two SGD solutions θ 1 and θ 2 , permute θ 1 and report the barrier between permuted θ 

Revised: In order to reduce the search space, here we only take two SGD solutions θ 1 and θ 2 , permute θ 1 and report the barrier between permuted θ

Original: and θ 2 as found by SA with n = 2 .
Revised: and θ 2 as found by SA with n = 2 .

Original: Figure 13 and Figure 7 show the effect of width and depth on barrier similarity between S and S (cid:48) before and after permutation.
Revised: Figure 13 and Figure 7 show the effect of width and depth on barrier similarity between S and S (cid:48) before and after permutation.

Original: Comparing Figure 13 and Figure 7 shows that SA succeeds in barrier removal.
Revised: Comparing Figure 13 and Figure 7 shows that SA succeeds in barrier removal.

Original: SA performance on S and S (cid:48) yields similar results for both before and after permutation scenarios.
Revised: SA performance on S and S (cid:48) yields similar results for both before and after permutation scenarios.

Original: Such similar performance is observed along a wide range of width and depth for both MLP and Shallow-CNN over different datasets (MNIST, SVHN, CIFAR10, CIFAR100).
Revised: Such similar performance is observed along a wide range of width and depth for both MLP and Shallow-CNN over different datasets (MNIST, SVHN, CIFAR10, CIFAR100).

Original: We look into effects of changing model size in terms of width and depth as in earlier sections, and note that similar trends hold for before and after permuting solution θ 1 .
Revised: We look into effects of changing model size in terms of width and depth as in earlier sections, and note that similar trends hold for before and after permuting solution θ 1 .

Original: Comparing Figure 13 and Figure 7 shows that reducing the search space makes SA more successful in ﬁnding the permutation { π } to remove the barriers.
Revised: Comparing Figure 13 and Figure 7 shows that reducing the search space makes SA more successful in ﬁnding the permutation { π } to remove the barriers.

Original: Speciﬁcally, SA can indeed ﬁnd permutations across different networks and datasets that result in zero barrier when applied to θ 1 .
Revised: Speciﬁcally, SA can indeed ﬁnd permutations across different networks and datasets that result in zero barrier when applied to θ 1 .

Original: SA can also ﬁnd permutations that reduce the barrier for both MLP and Shallow-CNN across different width, depth and datasets.
Revised: SA can also ﬁnd permutations that reduce the barrier for both MLP and Shallow-CNN across different width, depth and datasets.

Original: For example, such cases include MLP for MNIST, SVHN, CIFAR10, and CIFAR100 where depth is 1 and width is 2 3 and 2 4 ,
Revised: For example, such cases include MLP for MNIST, SVHN, CIFAR10, and CIFAR100 where depth is 1 and width is 2 3 and 2 4 ,

Original: MLP for MNIST where depth is 2 and width is 2 10 , Shallow-CNN for MNIST, SVHN, CIFAR10,
Revised: MLP for MNIST where depth is 2 and width is 2 10 , Shallow-CNN for MNIST, SVHN, CIFAR10,

Original: CIFAR100 where depth is 2 and width is 2 4 and for Shallow-CNN for MNIST where depth is 2 and width is 2 6 .
Revised: CIFAR100 where depth is 2 and width is 2 4 and for Shallow-CNN for MNIST where depth is 2 and width is 2 6 .

Original: [Figure]
Revised: [Figure]

Original: A. S IMULATED A NNEALING
Revised: A. S IMULATED A NNEALING

Original: We use Simanneal 4 as python module for simulated annealing.
Revised: We use Simanneal 4 as python module for simulated annealing.

Original: The process involves:
Revised: The process involves:

Original: • Randomly move or alter the state (generate a permutation) •
Revised: • Randomly move or alter the state (generate a permutation) •

Original: Assess the energy of the new state (permuted model) using the objective function (Linear Mode Connectivity based on Equation 1) • Compare the energy to the previous state and decide whether to accept the new solution or reject it based on the current temperature.
Revised: Assess the energy of the new state (permuted model) using the objective function (Linear Mode Connectivity based on Equation 1) • Compare the energy to the previous state and decide whether to accept the new solution or reject it based on the current temperature.

Original: For a move to be accepted, it must meet one of two requirements:
Revised: For a move to be accepted, it must meet one of two requirements:

Original: • The move causes a decrease in state energy (i
Revised: • The move causes a decrease in state energy (i

Original: e. an improvement in the objective function • The move increases the state energy (i.e. a slightly worse solution) but is within the bounds of the temperature.
Revised: e. an improvement in the objective function) • The move increases the state energy (i.e. a slightly worse solution) but is within the bounds of the temperature.

Original: Temperature .
Revised: Temperature .

Original: In each step, the generated permutation is chosen with the probability of P =
Revised: In each step, the generated permutation is chosen with the probability of P =

Original: e − cost temperature , where cost is the barrier at α = 12 .
Revised: e − cost temperature , where cost is the barrier at α = 12 .

Original: In the ﬁrst steps, as the temperature is high, there is a high probability that the worse neighbor is also selected.
Revised: In the ﬁrst steps, as the temperature is high, there is a high probability that the worse neighbor is also selected.

Original: The neighbor is another permutation that if applied, differs slightly in the order of the neurons/channels.
Revised: The neighbor is another permutation that if applied, differs slightly in the order of the neurons/channels.

Original: As we move forward the temperature decreases with T =
Revised: As we move forward the temperature decreases with T =

Original: e e − Tmax × steps Tmin × steps and we stick to permutations that improve the barrier.
Revised: e e − Tmax × steps Tmin × steps and we stick to permutations that improve the barrier.

Revised: Scaling the computation for SA .

Revised: In an experiment, we scale number of steps in simulated annealing to investigate the effect of this hyper-parameter.

Revised: If the barrier continues to decrease as the amount of computation increases and does not plateau, then this would suggest that with enough computation, the barrier found by simulated annealing could eventually go to zero.

Revised: Figure 14 shows that increasing number of steps exponentially, helps SA to ﬁnd better solutions.

Revised: Table 3 shows that as the number of steps increases ( 10 × ), ∆ moves towards 2 i.e. , 50% reduction in barrier ( ∆ > 0 means barrier does not plateau) .

Original: Our experiments show that the number of steps has the highest importance and we set it to 50K. (lower numbers decrease the producibility of the found permutations.).
Revised: Running SA for 50K steps takes 10K seconds on an n1-standard-8 GCP machine ( vCPU, 30 GB RAM) with 1xV100 GPU. Due to limited computational resources, we set number of the steps to 50K. [Table] [Figure]

Revised: [Table]

Original: The following code runs simulated annealing to ﬁnd the best permutation in Section4.
Revised: The following code runs simulated annealing to ﬁnd the best permutation in Section

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  B P ROOF OF T HEOREM
Revised: cost B I MPROVE S EARCH A LGORITHM

Revised: In the main text we use simulated annealing (SA) to ﬁnd the winning permutation π .

Revised: Figure 6 shows that SA is only able to reduce the barrier, and reducing search space (n = 2) helps in ﬁnding better permutations.

Revised: However, a critical question is still remained: Is there any better algorithm, that ﬁnds better permutations?

Revised: He et al.

Revised: (2018) proposed an algorithms that merges correlated, pre-trained deep neural networks for cross-model compression.

Revised: Their objective is to zip two neural networks, optimized for two different tasks, into one network.

Revised: The ultimate network does both tasks without losing too much accuracy on each task.

Revised: Their algorithm is based on layer-wise neuron sharing, which uses f(weights, post activations) to ﬁnd which neurons could be zipped together.

Revised: They deﬁne the similarity (or equivalently difference) of two neurons as below:

Revised: [Equation]

Revised: We also used their "functional difference" as a measure for neuron matching between two randomly initialized trained networks.

Revised: They used post activation as an approximation for Hessian matrices.

Revised: Calculating the difference between each pair of neurons based on Equation 2, gives an m × m matrix (m is width of the network).

Revised: In the second step, neurons with minimum distance are matched together in a greedy way i.e. , if n i,A and n j,B have the minimum distance, row i and column j is removed from the distance matrix.

Revised: Figure 15 shows that using functional difference, the barrier could be improved.

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: C M AKING E NSEMBLES

Revised: As stated before, our conjecture has implications for ensemble methods.

Revised: If two solutions lie at the periphery of a wide and ﬂat low loss region (a basin where models are linearly connected to each other), then ensembling them in their weight space (averaging), creates a model tending to the center of the region, which leads to performance improvement.

Revised: However, Simulated Annealing could not ﬁnd the optimal permutations for all cases.

Revised: Section B shows that using matching algorithms like functional difference could help to ﬁnd better permutations.

Revised: Motivated to create ensembles, Wortsman et al.

Revised: (2021) start with two (or more) random initializations and learn a subspace (a line or simplex) connecting them.

Revised: Throughout training they sample one (or more) points on this line and add the loss at this point to the loss of training.

Revised: In order to enforce diversity in fucntion space, they also add a regularization term as cosine similarity of two endpoints of the line.

Revised: However as they enforce two models to be in a line, the functional diversity of the ﬁnal ensemble is limited compare to our methods.

Revised: Here we combine their method with functional difference to make best of both worlds.

Revised: In this experiment, we train two randomly initialized networks separately, and then use functional difference to decrease the barrier between ﬁnal solutions.

Revised: Then we use learning subspace method to make them in one basin.

Revised: Our results on MLP with one hidden layer and width of 1024 neurons on

Revised: MNIST and CIFAR10 shows that this method outperforms the others.

Revised: [Table]

Revised: D P ROOF OF T HEOREM 3.

Original: We ﬁrst recap Theorem 1 below for convenience and then provide the proof
Revised: We ﬁrst recap Theorem 3.1 below for convenience and then provide the proof

Original: Theorem 2 (1) .
Revised: Theorem D.1 (3.1) .

Original: Let h be the number of hidden units, d be the input size.
Revised: Let h be the number of hidden units, d be the input size.

Original: Let the function f v , U ( x ) = v (cid:62) σ ( Ux ) where σ ( · ) is ReLU activation, v ∈ R h and U ∈ R h × d are parameters and x ∈ R
Revised: Let the function f v , U ( x ) = v (cid:62) σ ( Ux ) where σ ( · ) is ReLU activation, v ∈ R h and U ∈ R h × d are parameters and x ∈ R

Original: d is the input.
Revised: d is the input.

Original: We show that if each element of U and U (cid:48) is sampled uniformly from [ − 1 / √ d, 1 / √ d ] and each element of v and v (cid:48) is sampled uniformly from [ − 1 / √ h, 1 / √ h ] , then for any x ∈ R d such that (cid:107) x (cid:107) 2 = √ d , with probability 1 − δ over U , U (cid:48) , v , v (cid:48) , there exist a permutation such that
Revised: We show that if each element of U and U (cid:48) is sampled uniformly from [ − 1 / √ d, 1 / √ d ] and each element of v and v (cid:48) is sampled uniformly from [ − 1 / √ h, 1 / √ h ] , then for any x ∈ R d such that (cid:107) x (cid:107) 2 = √ d , with probability 1 − δ over U , U (cid:48) , v , v (cid:48) , there exist a permutation such that

Original: [Equation]
Revised: [Equation]

Original: 1 2 d +4 ) where v (cid:48)(cid:48) and U (cid:48)(cid:48) are permuted versions of v (cid:48) and U (cid:48) .
Revised: [Table] where v (cid:48)(cid:48) and U (cid:48)(cid:48) are permuted versions of v (cid:48) and U (cid:48) .

Original: Proof.
Revised: Proof.

Original: For any given ξ > 0 , we consider the set S ξ = {− 1 / √ d + ξ, − 1 / √ d + 3 ξ, . . .
Revised: For any given ξ > 0 , we consider the set S ξ = {− 1 / √ d + ξ, − 1 / √ d + 3 ξ, . . .

Original: , 1 / √ d − ξ } d which has size ( 1 ξ √ d ) d 5 .
Revised: , 1 / √ d − ξ } d which has size ( 1 ξ √ d ) d 5 .

Original: For any s ∈ S ξ , let C s ( U ) be the set of indices of rows of U that are closest in Euclidean distance to s than any other element in S ξ :
Revised: For any s ∈ S ξ , let C s ( U ) be the set of indices of rows of U that are closest in Euclidean distance to s than any other element in S ξ :

Original: [Equation]
Revised: [Equation]

Original: where for simplicity we assume that arg min returns a single element.
Revised: where for simplicity we assume that arg min returns a single element.

Original: We next use the function C s to specify a permutation that allows each row in U (cid:48) to be close to its corresponding row in U .
Revised: We next use the function C s to specify a permutation that allows each row in U (cid:48) to be close to its corresponding row in U .

Original: For every s ∈ S ξ , we consider a random matching of elements in C s ( U ) and C s ( U (cid:48) ) and when the sizes don’t match, add the extra items in U and U (cid:48) to the sets
Revised: For every s ∈ S ξ , we consider a random matching of elements in C s ( U ) and C s ( U (cid:48) ) and when the sizes don’t match, add the extra items in U and U (cid:48) to the sets

Original: I and I (cid:48) accordingly to deal with them later.
Revised: I and I (cid:48) accordingly to deal with them later.

Original: Since each element of U and U (cid:48) is sampled uniformly from [ − 1 / √ d, 1 / √ d ] , for each row in U and U (cid:48) , the probability of being assigned to each s ∈ S ξ is a multinomial distribution with equal probability for each s .
Revised: Since each element of U and U (cid:48) is sampled uniformly from [ − 1 / √ d, 1 / √ d ] , for each row in U and U (cid:48) , the probability of being assigned to each s ∈ S ξ is a multinomial distribution with equal probability for each s .

Original: Given any s ∈ S ξ , we can use Hoeffding’s inequality to bound the size of | C s ( U ) | with high probability.
Revised: Given any s ∈ S ξ , we can use Hoeffding’s inequality to bound the size of | C s ( U ) | with high probability.

Original: For any t ≥ 0 :
Revised: For any t ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: By union bound over all rows of U and U (cid:48) , with probability 1 − δ/ 3 , we have that for every s ∈ S ξ ,
Revised: By union bound over all rows of U and U (cid:48) , with probability 1 − δ/ 3 , we have that for every s ∈ S ξ ,

Original: [Equation]
Revised: [Equation]

Original: Consider I and I (cid:48) which are the sets of indices that we throw out during the index assignment because of the size mismatch.
Revised: Consider I and I (cid:48) which are the sets of indices that we throw out during the index assignment because of the size mismatch.

Original: Then, based on above inequality, we have that with probability 1 − δ/ 3 ,
Revised: Then, based on above inequality, we have that with probability 1 − δ/ 3 ,

Original: [Equation]
Revised: [Equation]

Original: We next randomly match the indices in I and I .
Revised: We next randomly match the indices in I and I .

Original: Let U (cid:48)(cid:48) be the matrix after applying the permutation to U (cid:48) that corresponds to above matching of rows of U (cid:48) to their corresponding row in U .
Revised: Let U (cid:48)(cid:48) be the matrix after applying the permutation to U (cid:48) that corresponds to above matching of rows of U (cid:48) to their corresponding row in U .

Original: Note that 
Revised: Note that for any i ∈ [ h ] \ I , we have that (cid:13)(cid:13) u i − u (cid:48)(cid:48) i (cid:13)(cid:13) ∞ ≤ 2 ξ and for i ∈

Original: We next upper bound the left hand side of the inequality in the theorem statement:
Revised: I , we have (cid:13)(cid:13) u i − u (cid:48)(cid:48) i (cid:13)(cid:13) ∞ ≤ 2 / [Equation]

Revised: d

Original: [Equation]
Revised: [Equation]

Revised: We next upper bound the left hand side of the inequality in the theorem statement:

Original:  Since each element of v is sampled uniformly from [ − 1 / √h, 1 / √ h]
Revised: [Equation] = [Equation] Since each element of v is sampled uniformly from [ − 1 / √ h

Original: [Equation]
Revised: [Equation]

Original:  for any r ∈ R h we have that E [ v (cid:62) r ] = 0 and by Hoeffding’s inequality,
Revised: 1 / √ h ] [Equation] for any r ∈ R h we have that E [ v (cid:62) r ] = 0 and by Hoeffding’s inequality,

Original: [Equation]
Revised: [Equation]

Original: Using the above argument

Original: [Equation]
Revised: [Equation]

Original:  with probability 1 − δ/
Revised: Using the above argument, with probability 1 − δ/

Original: [Equation]
Revised: [Equation]

Original: we can bound the right hand side of inequality (6) as follows:
Revised: we can bound the right hand side of inequality (7) as follows:

Original: [Equation]
Revised: [Equation]

Original: (10) where the inequality 9 is due to Lipschitz property of ReLU activations.
Revised: where the inequality 10 is due to Lipschitz property of ReLU activations.

Original: Now, all we need to do is to bound (cid:13)(cid:13)
Revised: Now, all we need to do is to bound (cid:13)(cid:13)

Original: ( U − U (cid:48)(cid:48) )
Revised: ( U − U (cid:48)(cid:48) )

Original: x (cid:13)(cid:13) 2 .
Revised: x (cid:13)(cid:13) 2 .

Original: Note that for any ( i, j ) ∈
Revised: Note that for any ( i, j ) ∈

Original: [ h ] × [ d ] , u ij − u (cid:48)(cid:48) ij is an independent random variable with mean zero and bounded magnitude( 2 / √ d if i ∈ I and 2 ξ otherwise).
Revised: [ h ] × [ d ] , u ij − u (cid:48)(cid:48) ij is an independent random variable with mean zero and bounded magnitude( 2 / √ d if i ∈ I and 2 ξ otherwise).

Original: Therefore, we can again use the Hoffding’s inequality similar to inequality (7) for each row i and after taking a union bound, we have the following inequality with probability 1 − δ/ 3 ,
Revised: Therefore, we can again use the Hoffding’s inequality similar to inequality (8) for each row i and after taking a union bound, we have the following inequality with probability 1 − δ/ 3 ,

Original: [Equation]
Revised: [Equation]

Original: Where the last inequality is using (cid:107) x (cid:107) 2 = √ d .
Revised: Where the last inequality is using (cid:107) x (cid:107) 2 = √ d .

Original: Substituting the above inequality into the right hand side of the inequality (10), gives us the following upper bound on the left hand side of the inequality in the theorem statement:
Revised: Substituting the above inequality into the right hand side of the inequality (11), gives us the following upper bound on the left hand side of the inequality in the theorem statement:

Original: [Equation]
Revised: [Equation]

Original: Setting ξ =
Revised: Setting ξ =

Original: (cid:15)/ (cid:112) 4 d log(12 /δ )
Revised: (cid:15)/ [Equation]

Original:  log(12 h/δ ) , gives the following bound on h :
Revised: (12 /δ ) log(12 h/δ ) , gives the following bound on h :

Original: [Equation]
Revised: [Equation]

Original: Therefore [Equation] we have:
Revised: Therefore, we have:

Original: [Equation]
Revised: [Equation]

Original: Using the above inequality [Equation] we have (cid:15) =
Revised: Using the above inequality, we have (cid:15) =

Original: ˜ O ( h − 1 2 d +4 ) .
Revised: ˜ O ( h − 1 2 d +4 ) .

Revised: E A DDITIONAL PLOTS E. B ARRIER BEHAVIOR UNDER NOISY LABELS

Revised: Related works (Zhang et al., 2017) show that neural nets can memorize random labels.

Revised: In this section we want to see whether the barrier changes if one starts to inject random labels into the training dataset.

Revised: Our results over 5 different runs show that the barrier size behavior does not change, however including higher level of noise in labels lead to small increase in barrier size.

Original: C A DDITIONAL PLOTS C. B ARRIER : VGG AND R ES N ET
Revised: [Figure] E. B ARRIER : VGG AND R ES N ET

Original: Figure 14 shows the barrier similarity for  and  families between real world and our model.
Revised: Figure 17 shows the barrier similarity for VGG and ResNet families between real world and our model.

Original: The left  panels shows the effect of width, while the right  panels illustrate the depth.
Revised: The left two panels shows the effect of width [Figure] while the right two panels illustrate the depth.

Original: As discussed in section 2, the barrier for VGGs and ResNets, for both real world and our model, is saturated at a high value and does not change.
Revised: As discussed in section [Figure] the barrier for VGGs and ResNets, for both real world and our model, is saturated at a high value and does not change.

Original: [Figure]
Revised: [Figure]

Original: C. S MALL NETWORKS
Revised: E. S MALL NETWORKS

Original: We consider MLPs with the same architecture starting from 100 different initializations and trained on the MNIST dataset.
Revised: We consider MLPs with the same architecture starting from 100 different initializations and trained on the MNIST dataset.

Original: For each pair of the networks we calculate their loss barrier and plot the histogram on the values.
Revised: For each pair of the networks we calculate their loss barrier and plot the histogram on the values.

Original: Next for each pair of the networks, we ﬁnd the permutation that minimizes the barrier between them and plot the histogram for all the pairs.
Revised: Next for each pair of the networks, we ﬁnd the permutation that minimizes the barrier between them and plot the histogram for all the pairs.

Original: We do this investigation for different network sizes.
Revised: We do this investigation for different network sizes.

Original: The results are shown in Figure 15 top row.
Revised: The results are shown in Figure 18 top row.

Original: Note that since we consider all possible pairs, the observed barrier values are not i.i.d.
Revised: Note that since we consider all possible pairs, the observed barrier values are not i.i.d.

Original: If instead we randomly divide the 100 trained networks [Figure] p r o b a b ili t y [Figure]
Revised: If instead we randomly divide the 100 trained networks into two sets and choose pairs that are made by picking one network from each set, we will have an i.i.d sampling strategy.

Original: [Figure]
Revised: We investigate the pairs of networks trained on MNIST and measure the value of the direct and indirect barriers between them

Original: [Figure]
Revised: [Table]

Original: barrier    p r o b a b ili t y 4 hidden units 8 hidden units 16 hidden units 32 hidden units 64 hidden units

Original: [Figure]
Revised: [Table]

Original: i n d i r e c t b a rr i e r

Original: [Figure]
Revised: [Equation]

Original: into two sets and choose pairs that are made by picking one network from each set, we will have an i.i.d sampling strategy. We investigate the pairs of networks trained on MNIST and measure the value of the direct and indirect barriers between them. Indirect barrier between two networks A, C is minimum over all possible intermediate points B of maximum of barrier between A, B and barrier between B, C, i.e. ,
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  The reason we look into this value is that if maximum between two barriers is small, it means that both barriers are small and therefore there exist an indirect path between A and C.
Revised: [Table] The reason we look into this value is that if maximum between two barriers is small, it means that both barriers are small and therefore there exist an indirect path between A and C.

Original: C. L OSS BARRIERS ON THE TEST SET
Revised: E. L OSS BARRIERS ON THE TEST SET

Original: Width .
Revised: Width .

Original: We evaluate the impact of width on the test barrier size in Figure 16.
Revised: We evaluate the impact of width on the test barrier size in Figure 19.

Original: In comparison to
Revised: In comparison to

Original: Figure 2 the magnitude of test barriers are shifted to lower values as the test accuracy is lower than train accuracy.
Revised: Figure 2 the magnitude of test barriers are shifted to lower values as the test accuracy is lower than [Figure] p r o b a b ili t y [Figure]

Revised: [Figure]

Revised: p r o b a b ili t y 4 hidden units 8 hidden

Original: This effect is intensiﬁed for harder tasks such as CIFAR100.
Revised: [Figure] [Figure] [Figure] units 64 hidden

Revised: [Figure]

Revised: i n d i r e c t b a rr i e r

Revised: [Figure]

Revised: train accuracy.

Revised: This effect is intensiﬁed for harder tasks such as CIFAR100.

Original: The double descent phenomena is also observed here, especially for simpler tasks, e.g. , MNIST and SVHN.
Revised: The double descent phenomena is also observed here, especially for simpler tasks, e [Figure] g. , MNIST and SVHN [Figure]

Original: Depth .
Revised: [Figure] [Figure] [Figure] ResNet

Revised: MNISTSVHNCIFAR10CIFAR

Original:  We evaluate the impact of depth on the test barrier size in Figure 17.
Revised: Depth . We evaluate the impact of depth on the test barrier size in Figure 20.

Original: For MLPs, we ﬁxed the layer width at 2 10 while adding identical layers as shown along the x-axis.
Revised: For MLPs, we ﬁxed the layer width at 2 10 while adding identical layers as shown along the x-axis.

Original: Similar to Figure 3 we observe a fast and signiﬁcant barrier increase as more layers are added.
Revised: Similar to Figure 3 we observe a fast and signiﬁcant barrier increase as more layers are added.

Original: In comparison to Figure 3 the magnitude of test barriers are shifted to lower values as the test accuracy is lower than train accuracy.
Revised: In comparison to Figure 3 the magnitude of test barriers are shifted to lower values as the test accuracy is lower than train accuracy.

Original: C. S IMILARITY OF S AND S (cid:48) ON THE TEST SET
Revised: E. S IMILARITY OF S AND S (cid:48) ON THE TEST SET

Original: We note SA success on test barrier removal by comparing Figure 18 and Figure 19.
Revised: We note SA success on test barrier removal by comparing Figure 21 and Figure 22.

Original: SA performance on S and S (cid:48) yields similar results for both before and after permutation scenarios.
Revised: SA performance on S and S (cid:48) yields similar results for both before and after permutation scenarios.

Original: Such similar performance is observed along a wide range of width and depth for both MLP and Shallow-CNN over different datasets(MNIST, SVHN, CIFAR10, CIFAR100).
Revised: Such similar performance is observed along a wide range of width and depth for both MLP and Shallow-CNN over different datasets(MNIST, SVHN, CIFAR10, CIFAR100).

Original: We look into effects of changing
Revised: We look into effects of changing

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: model size in terms of width and depth as in earlier Sections, and note that similar trends hold for before and after permuting solution θ
Revised: model size in terms of width and depth as in earlier Sections, and note that similar trends hold for before and after permuting solution θ

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]