Original: Random initialisations performing above chance and how to find them
Revised: Random initialisations performing above chance and how to find them

Original: author names withheld
Revised: ROMEIER INF ETHZ VOSWALDJ @ ETHZ

Revised: YASSIR .

Original: Under Review for OPT
Revised: AKRAM @ INF ETHZ NZUCCHET

Original:  Abstract
Revised: INF ETHZ STEGER Abstract

Original: Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions.
Revised: Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions.

Original: Entezari et al.
Revised: Entezari et al.

Original: recently conjectured that despite different initialisations, the solutions found by SGD lie in the same loss valley after taking into account the permutation invariance of neural networks.
Revised: recently conjectured that despite different initialisations, the solutions found by SGD lie in the same loss valley after taking into account the permutation invariance of neural networks.

Original: Concretely, they hypothesise that any two solutions found by SGD can be permuted such that the linear interpolation between their parameters forms a path without significant increases in loss.
Revised: Concretely, they hypothesise that any two solutions found by SGD can be permuted such that the linear interpolation between their parameters forms a path without significant increases in loss.

Original: Here, we use a simple but powerful algorithm to find such permutations that allows us to obtain direct empirical evidence that the hypothesis is true in fully connected networks.
Revised: Here, we use a simple but powerful algorithm to find such permutations that allows us to obtain direct empirical evidence that the hypothesis is true in fully connected networks.

Original: Strikingly, we find that two networks already live in the same loss valley at the time of initialisation and averaging their random, but suitably permuted initialisation performs significantly above chance.
Revised: Strikingly, we find that two networks already live in the same loss valley at the time of initialisation and averaging their random, but suitably permuted initialisation performs significantly above chance.

Original: In contrast, for convolutional architectures, our evidence suggests that the hypothesis does not hold.
Revised: In contrast, for convolutional architectures, our evidence suggests that the hypothesis does not hold.

Original: Especially in a large learning rate regime,
Revised: Especially in a large learning rate regime,

Original: SGD seems to discover diverse modes.
Revised: SGD seems to discover diverse modes.

Original: 1. Introduction
Revised: 1. Introduction

Original: Neural networks trained from different initialisations typically have similar decision boundaries [34], their predictions agree on a large fraction of data [25] and even their internal representations are similar [17, 22].
Revised: Neural networks trained from different initialisations typically have similar decision boundaries [34], their predictions agree on a large fraction of data [25] and even their internal representations are similar [17, 22].

Original: This raises the question if all trained instances of neural networks are created equal.
Revised: This raises the question if all trained instances of neural networks are created equal.

Original: Entezari et al.
Revised: Entezari et al.

Original: [8] recently proposed one way to formalise this idea and hypothesised that any two solutions found by stochastic gradient descent (SGD) live in the same loss valley when accounting for the permutation invariance of neural networks.
Revised: [8] recently proposed one way to formalise this idea and hypothesised that any two solutions found by stochastic gradient descent (SGD) live in the same loss valley when accounting for the permutation invariance of neural networks.

Original: More precisely, they suggest the following hypothesis.
Revised: More precisely, they suggest the following hypothesis.

Original: Hypothesis 1 (All SGD solutions are created equal, Entezari et al.
Revised: Hypothesis 1 (All SGD solutions are created equal, Entezari et al.

Original: For two solutions θ 1 , θobtained by SGD, there is a permuted instance of the first solution π ( θ 1 ) so that there is no (significant) increase in loss along the linear interpolation between π ( θ 1 ) and θ 2 .
Revised: For two solutions θ 1 , θobtained by SGD, there is a permuted instance of the first solution π ( θ 1 ) so that there is no (significant) increase in loss along the linear interpolation between π ( θ 1 ) and θ 2 .

Original: If Hypothesis 1 is true, this would greatly improve our understanding of the loss landscape of neural networks, likely impact ensemble learning [12] as well as network sparsification [10] and it would potentially allow to improve optimisation and generalisation [15] in neural networks.
Revised: If Hypothesis 1 is true, this would greatly improve our understanding of the loss landscape of neural networks, likely impact ensemble learning [12] as well as network sparsification [10] and it would potentially allow to improve optimisation and generalisation [15] in neural networks.

Original: Here, we investigate this hypothesis by explicitly constructing permutations with the desired properties, leading to a range of novel findings that elucidate the structure of neural loss landscapes.
Revised: Here, we investigate this hypothesis by explicitly constructing permutations with the desired properties, leading to a range of novel findings that elucidate the structure of neural loss landscapes

Original: • We find strong, direct evidence in support of the hypothesis for fully connected networks of moderate and large width on standard datasets.
Revised: • We find strong, direct evidence in support of the hypothesis for fully connected networks of moderate and large width on standard datasets.

Original: [Figure]

Original: • For fully connected networks, we find that an even stronger version of the hypothesis is true: The networks already live in the same loss valley at initialisation.
Revised: • For fully connected networks, we find that an even stronger version of the hypothesis is true: The networks already live in the same loss valley at initialisation.

Original: In particular, averaging the random initialisations π ( θ init 1 ) , θ initusing the permutation revealed through training, gives a network performing significantly above chance.
Revised: In particular, averaging the random initialisations π ( θ init 1 ) , θ initusing the permutation revealed through training, gives a network performing significantly above chance.

Original: • For convolutional neural networks (CNNs), our evidence suggests that there is a notable loss barrier between solutions also after accounting for permutation invariance.
Revised: • For convolutional neural networks (CNNs), our evidence suggests that there is a notable loss barrier between solutions also after accounting for permutation invariance.

Original: In particular, and interestingly, in the large learning rate regime, SGD seems to explore a genuinely diverse set of solutions.
Revised: In particular, and interestingly, in the large learning rate regime, SGD seems to explore a genuinely diverse set of solutions.

Original: We discuss exciting concurrent work [1] as well as generally related work on permutation invariance and mode connectivity in Appendix A .
Revised: We discuss exciting concurrent work [1] as well as generally related work on permutation invariance and mode connectivity in Appendix A .

Original: 2. Background
Revised: 2. Background

Original: We would like to evaluate whether two sets of parameters θ 1 , θ 2 lie in the same loss valley after accounting for permutation invariance.
Revised: We would like to evaluate whether two sets of parameters θ 1 , θ 2 lie in the same loss valley after accounting for permutation invariance.

Original: To this end, we seek a permutation π ∗ of the parameters θthat minimizes the loss barrier between the two sets of parameters after applying the permutation, i.e. we would like to solve
Revised: To this end, we seek a permutation π ∗ of the parameters θthat minimizes the loss barrier between the two sets of parameters after applying the permutation, i.e. we would like to solve

Original: [Equation]
Revised: [Equation]

Original: While this problem is computationally intractable, a simple heuristic allows us to find good permutations in practice.
Revised: While this problem is computationally intractable, a simple heuristic allows us to find good permutations in practice.

Original: We relax the problem by considering the neuron activations of each layer separately.
Revised: We relax the problem by considering the neuron activations of each layer separately.

Original: For each layer, we then find a permutation that makes the resulting neuron activations as similar as possible.
Revised: For each layer, we then find a permutation that makes the resulting neuron activations as similar as possible.

Original: Formally, given a set of d data points, the neuron activations of the networks parameterized by θ 1 and θ 2 are given by the matrices A, B ∈ R d × n , where n denotes the number of neurons in the hidden layer (or the number of channels for CNNs).
Revised: Formally, given a set of d data points, the neuron activations of the networks parameterized by θ 1 and θ 2 are given by the matrices A, B ∈ R d × n , where n denotes the number of neurons in the hidden layer (or the number of channels for CNNs).

Original: We define a similarity function k ( · , · ) which measures for any pair of neurons i, j how similar their activation vectors A i , B j are and specify our surrogate objective as finding the permutation π which maximizes the summed similarity between all matched neuron activations s (
Revised: We define a similarity function k ( · , · ) which measures for any pair of neurons i, j how similar their activation vectors A i , B j are and specify our surrogate objective as finding the permutation π which maximizes the summed similarity between all matched neuron activations s (

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Figure]

Original: Maximizing s ( π ) corresponds to finding a maximum matching in a bipartite graph and can be solved efficiently using the Hungarian algorithm.
Revised: Maximizing s ( π ) corresponds to finding a maximum matching in a bipartite graph and can be solved efficiently using the Hungarian algorithm.

Original: We explored different variants of this idea which all performed similarly.
Revised: We explored different variants of this idea which all performed similarly.

Original: Unless noted otherwise we use the cosine similarity as similarity function k and consider the neuron activations after the non-linearity.
Revised: Unless noted otherwise we use the cosine similarity as similarity function k and consider the neuron activations after the non-linearity.

Original: Finding permutations based on this idea has already been explored in [5, 22] and re-described e.g. in [1, 33, 35, 38].
Revised: Finding permutations based on this idea has already been explored in [5, 22] and re-described e.g. in [1, 33, 35, 38].

Original: 3. Results 3.1. Fully connected networks
Revised: 3. Results 3.1. Fully connected networks

Original: For the first set of experiments, we will focus on fully-connected networks with varying width and depth trained on MNIST [21], Fashion MNIST [37] or CIFAR 10 [19].
Revised: For the first set of experiments, we will focus on fully-connected networks with varying width and depth trained on MNIST [21], Fashion MNIST [37] or CIFAR 10 [19].

Original: In the main paper we will show results on Fashion MNIST; results on other datasets are qualitatively similar and can be found in Appendix G.
Revised: In the main paper we will show results on Fashion MNIST; results on other datasets are qualitatively similar and can be found in Appendix G.

Original: The training and evaluation procedure is simple and outlined in Figure 1(a).
Revised: The training and evaluation procedure is simple and outlined in Figure 1(a).

Original: We train two networks independently, for 5 epochs each with AdamW [16, 23] with PyTorch [29] default settings.
Revised: We train two networks independently, for 5 epochs each with AdamW [16, 23] with PyTorch [29] default settings.

Original: We then compute the permutation making the networks’ activations as similar as possible.
Revised: We then compute the permutation making the networks’ activations as similar as possible.

Original: Subsequently, we evaluate networks along the linear interpolation between the two networks, either using their unpermuted or permuted versions.
Revised: Subsequently, we evaluate networks along the linear interpolation between the two networks, either using their unpermuted or permuted versions.

Original: Trained networks lie in the same loss valley after permutation.

Original: Figure 2 shows the results of these experiments.

Original: The loss barrier between solutions decreases dramatically after accounting for permutation invariance.

Original: This decrease confirms that permutation invariance is important for understanding mode connectivity.

Original: Furthermore, by sufficiently increasing the width of the networks the loss barrier is eliminated completely by accounting for permutation invariance.

Original: Thus, in wide

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: T e s t
Revised: T e s t

Original: A cc .
Revised: A cc .

Original: [Figure]
Revised: [Figure]

Revised: Single Net

Original: Vanilla Avg.
Revised: Vanilla Avg.

Original: Perm Avg.
Revised: Perm Avg.

Original: [Figure] Width 
Revised: 2048 Width 0.

Original: [Figure]

Original: Hidden Layers:   2048 Width 0.
Revised: Hidden Layers:   2048 Width 0.

Original: Hidden Layers:
Revised: Hidden Layers:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: T r a i n
Revised: T r a i n

Original: N LL
Revised: N LL

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: Trained networks lie in the same loss valley after permutation.

Revised: Figure 2 shows the results of these experiments.

Revised: The loss barrier between solutions decreases dramatically after accounting for permutation invariance.

Revised: This decrease confirms that permutation invariance is important for understanding mode connectivity.

Revised: Furthermore, by sufficiently increasing the width of the networks the loss barrier is eliminated completely by accounting for permutation invariance.

Original:  networks, this directly confirms the hypothesis of Entezari et al.
Revised: Thus, in wide networks, this directly confirms the hypothesis of Entezari et al.

Original: [8], that there is no significant loss barrier between SGD solutions after taking permutation invariance into account.
Revised: [8], that there is no significant loss barrier between SGD solutions after taking permutation invariance into account.

Original: However, in narrower networks, we did not find permutations that eliminate the loss barrier suggesting that the two solutions are genuinely different even when accounting for permutation invariance.
Revised: However, in narrower networks, we did not find permutations that eliminate the loss barrier suggesting that the two solutions are genuinely different even when accounting for permutation invariance.

Original: That said, we cannot exclude the possibility that our permutations are suboptimal, and that “better” permutations would eliminate the loss barrier.
Revised: That said, we cannot exclude the possibility that our permutations are suboptimal, and that “better” permutations would eliminate the loss barrier.

Original: Indeed, a thorough falsification of the hypothesis in narrower networks would require an exhaustive (and therefore infeasible) search over the space of permutations.
Revised: Indeed, a thorough falsification of the hypothesis in narrower networks would require an exhaustive (and therefore infeasible) search over the space of permutations.

Original: Averaged, random initialisations perform above chance after permutations.
Revised: Averaged, random initialisations perform above chance after permutations.

Original: We have seen that trained networks usually live in the same, or at least similar, loss valleys when accounting for  permutation invariance.
Revised: We have seen that trained networks usually live in the same, or at least similar, loss valleys when accounting for [Figure] permutation invariance.

Original: Next, we ask how much training is required until the networks enter a shared loss valley.
Revised: Next, we ask how much training is required until the networks enter a shared loss valley.

Original: To investigate this, we conduct experiments as outlined in Figure 1(b), i.e. we take two random initialisations, train them with independently shuffled data, determine a permutation that makes the trained networks similar and then apply this permutation to the (untrained) initialisation.
Revised: To investigate this, we conduct experiments as outlined in Figure 1(b), i e. we take two random initialisations, train them with independently shuffled data, determine a permutation that makes the trained networks similar and then apply this permutation to the (untrained) initialisation.

Original: Figure 3(left) shows the interpolation between those permuted initialisations and shows a rather sur- [Figure]
Revised: Figure 3(left) shows the interpolation between those permuted initialisations and shows a rather surprising discovery:

Original: prising discovery: Already the initialisations live in a shared loss valley (up to a permutation), and simply averaging the permuted initialisations gives significantly above chance performance.
Revised: Already the initialisations live in a shared loss valley (up to a permutation), and simply averaging the permuted initialisations gives significantly above chance performance.

Original: Figure 3(mid) shows that performance of these random initialisations can be further improved by averaging more than two of them.
Revised: Figure 3(mid) shows that performance of these random initialisations can be further improved by averaging more than two of them.

Original: Figure 3(right) also shows that comparatively little training is required to reveal the shared loss valley between random initialisations.
Revised: Figure 3(right) also shows that comparatively little training is required to reveal the shared loss valley between random initialisations.

Original: We also note that it is possible to use the training data to find permutations that put initialisations into the same loss valley without actually training the networks, see Appendix B.
Revised: We also note that it is possible to use the training data to find permutations that put initialisations into the same loss valley without actually training the networks, see Appendix B.

Original: Summary.
Revised: Summary.

Original: Overall, these findings give a significantly improved understanding of the loss landscape of fully-connected networks.
Revised: Overall, these findings give a significantly improved understanding of the loss landscape of fully-connected networks.

Original: They reveal that, up to permutation, fully-connected feedforward networks live in the same loss valley at initialisation.
Revised: They reveal that, up to permutation, fully-connected feedforward networks live in the same loss valley at initialisation.

Original: Moreover, even as the networks start specialising during training, the loss barrier separating any two SGD solutions remains small. 
Revised: Moreover, even as the networks start specialising during training, the loss barrier separating any two SGD solutions remains small. [Figure]

Original: 3.2. Convolutional networks
Revised: 3.2. Convolutional networks

Original: Next, we investigate simple CNNs purely composed of convolutional layers with no skip connections and a linear readout layer (see Appendix F for network and training details).
Revised: Next [Figure] we investigate simple CNNs purely composed of convolutional layers with no skip connections and a linear readout layer (see Appendix F for network and training details).

Original: We again find that accounting for permutation invariance significantly reduces the loss barrier between SGD solutions, in a wide range of settings and as exemplified in Figure 4; however a[Figure] notable barrier remained in most of our settings, including wide networks.
Revised: We again find that accounting for permutation invariance significantly reduces the loss barrier between SGD solutions, in a wide range of settings and as exemplified in Figure 4; however a notable barrier remained in most of our settings, including wide networks.

Original: This is also consistent with results from more complex architectures, see Fig.
Revised: This is also consistent with results from more complex architectures, see Fig.

Original: 3 in [35].
Revised: 3 in [35].

Original: In addition [Figure] we found that the loss barrier in CNNs is significantly more sensitive to training details, like the learning rate.
Revised: In addition, we found that the loss barrier in CNNs is significantly more sensitive to training details, like the learning rate.

Original: 1 One example of this behaviour is shown in Figure 4 (left&mid), where with a larger learning rate the barrier between solutions increases.
Revised: 2 One example of this behaviour is shown in Figure 4 (left&mid), where with a larger learning rate the barrier between solutions increases.

Original: We emphasise that this behaviour can be observed across different convolutional architectures, datasets and also optimizers (AdamW as well as SGD+momentum).
Revised: We emphasise that this behaviour can be observed across different convolutional architectures, datasets and also optimizers (AdamW as well as SGD+momentum).

Original: In all these settings, the loss barrier increases for large learning rates, which are at the edge of training stability (occasionally, runs with these high learning rates fail to converge).
Revised: In all these settings, the loss barrier increases for large learning rates, which are at the edge of training stability (occasionally, runs with these high learning rates fail to converge).

Original: These results suggest that for CNNs trained with large learning rates, SGD explores diverse loss valleys, which differ also after accounting for permutation invariance.
Revised: These results suggest that for CNNs trained with large learning rates, SGD explores diverse loss valleys, which differ also after accounting for permutation invariance.

Original: Moreover, we note that for CNNs, different initialisations do not appear to come from a shared loss valley, see Figure 4 (right).
Revised: Moreover, we note that for CNNs, different initialisations do not appear to come from a shared loss valley, see Figure 4 (right).

Original: Summary.
Revised: Summary.

Original: These results reveal two interesting differences between fully-connected and convolutional architectures.
Revised: These results reveal two interesting differences between fully-connected and convolutional architectures.

Original: Firstly, in fully-connected networks already the initialisations live in a shared loss valley, while this does not seem to be the case for CNNs.
Revised: Firstly, in fully-connected networks already the initialisations live in a shared loss valley, while this does not seem to be the case for CNNs.

Original: Secondly, CNNs exhibit a richer training behaviour and in particular for large learning rates, close to training instability, they appear to explore genuinely different loss valleys, while solutions found by SGD in fully connected network are more similar. 
Revised: Secondly, CNNs exhibit a richer training behaviour and in particular for large learning rates, close to training instability, they appear to explore genuinely different loss valleys, while solutions found by SGD in fully connected network are more similar. Acknowledgements Simon Schug was supported by grant no. PZ00P3 186027 from the Swiss National Science Foundation.

Revised: Robert Meier was supported by grant no.

Revised: CRSII5 173721 of the Swiss National Science

Original:  Appendix A. Related work A.1. Concurrent related work
Revised: Foundation. Johannes von Oswald was funded by the Swiss Data Science Center (J.v.O. P18-03). Nicolas Zucchet and Yassir Akram were supported by ETH Research Grant no. ETH-23 21-1. Appendix A. Related work A.1. Concurrent related work

Original: Concurrently, Ainsworth et al.
Revised: Concurrently, Ainsworth et al.

Original: [1] explored a similar approach to investigate Hypothesis 1.
Revised: [1] explored a similar approach to investigate Hypothesis 1.

Original: In particular, the base algorithm to find permutations is analogous to ours and that in prior work [5, 22].
Revised: In particular, the base algorithm to find permutations is analogous to ours and that in prior work [5, 22].

Original: While both papers agree in their key findings, they also highlight different features.
Revised: While both papers agree in their key findings, they also highlight different features.

Original: The main differences are briefly described below:
Revised: The main differences are briefly described below:

Original: Ainsworth et al.
Revised: Ainsworth et al.

Original: [1] present results in a ResNet18 on CIFAR 10, showing that the loss barrier also decreases (and starts to disappear) with increasing width.
Revised: [1] present results in a ResNet18 on CIFAR 10, showing that the loss barrier also decreases (and starts to disappear) with increasing width.

Original: They also show a simple counterexample to the hypothesis for a small network with artificial data and explore merging networks trained on different subsets of the data.
Revised: They also show a simple counterexample to the hypothesis for a small network with artificial data and explore merging networks trained on different subsets of the data.

Original: In contrast, our contributions contains a systematic investigation into width- & depth-dependence of the loss barrier in fully-connected networks and we highlight the learning rate dependence of the loss barrier in CNNs.
Revised: In contrast, our contributions contains a systematic investigation into width- & depth-dependence of the loss barrier in fully-connected networks and we highlight the learning rate dependence of the loss barrier in CNNs.

Original: Moreover, we showcase that in fully connected networks, already the untrained initialisations live in the same loss valley.
Revised: Moreover, we showcase that in fully connected networks, already the untrained initialisations live in the same loss valley.

Original: A.2. Prior related work
Revised: A.2. Prior related work

Original: The question how similar different neural networks are has inspired research from a variety of angles.
Revised: The question how similar different neural networks are has inspired research from a variety of angles.

Original: For example, Li et al.
Revised: For example, Li et al.

Original: [22] train independent copies of CNNs and investigate how well individual neurons and groups of neurons from one network correspond to groups of different neurons; part of their methodology also relies on computing matchings analogously to our approach.
Revised: [22] train independent copies of CNNs and investigate how well individual neurons and groups of neurons from one network correspond to groups of different neurons; part of their methodology also relies on computing matchings analogously to our approach.

Original: Kornblith et al.
Revised: Kornblith et al.

Original: [17] investigates similarity of layer representations drawing on work from neuroscience (e.g. [18]) and machine learning [20].
Revised: [17] investigates similarity of layer representations drawing on work from neuroscience (e.g. [18]) and machine learning [20].

Original: Their similarity measure is theoretically wellmotivated and suggests that different networks learn considerably more similar representations than suggested by similarity measures of prior work [27, 30, 36].
Revised: Their similarity measure is theoretically well-motivated and suggests that different networks learn considerably more similar representations than suggested by similarity measures of prior work [27, 30, 36].

Original: Mehrer et al.
Revised: Mehrer et al.

Original: [26] also study network similarity with methods from neuroscience and additionally conduct insightful experiments on how inter- or intra- class variance contribute to differences between networks.
Revised: [26] also study network similarity with methods from neuroscience and additionally conduct insightful experiments on how inter- or intraclass variance contribute to differences between networks.

Original: All [17, 22, 26] find that representations agree very strongly in early layers and decay to some extent for deeper layers.
Revised: All [17, 22, 26] find that representations agree very strongly in early layers and decay to some extent for deeper layers.

Original: This is consistent with our findings in Figure 2 suggesting that deeper networks are separated by higher loss barriers.
Revised: This is consistent with our findings in Figure 2 suggesting that deeper networks are separated by higher loss barriers.

Original: In addition, our work is closely connected to mode connectivity.
Revised: In addition, our work is closely connected to mode connectivity.

Original: It was first shown theoretically that any two modes are connected by a (potentially complex, non-linear) path of low loss in networks with one hidden layer [11].
Revised: It was first shown theoretically that any two modes are connected by a (potentially complex, non-linear) path of low loss in networks with one hidden layer [11].

Original: This was confirmed in a range of empirical settings [7, 12].
Revised: This was confirmed in a range of empirical settings [7, 12].

Original: Tatro et al.
Revised: Tatro et al.

Original: [35] show that these paths can be simplified or their loss barrier lowered when taking permutation invariance into account.
Revised: [35] show that these paths can be simplified or their loss barrier lowered when taking permutation invariance into account.

Original: They find permutations similarly as us.
Revised: They find permutations similarly as us.

Original: Thus, their work is overall similar in spirit to ours, with the difference that we investigate linear paths between modes and how architecture and training choices affect mode connectivity.
Revised: Thus, their work is overall similar in spirit to ours, with the difference that we investigate linear paths between modes and how architecture and training choices affect mode connectivity.

Original: Theoretically, [4, 32] investigate the connectivity between permutations of a given mode.
Revised: Theoretically, [4, 32] investigate the connectivity between permutations of a given mode.

Original: We note that this is different from our objective to determine whether independently obtained modes are (approximately) permutations of one another.
Revised: We note that this is different from our objective to determine whether independently obtained modes are (approximately) permutations of one another.

Original: Additionally, there is work developing network-/kernel-representations which are invariant to permutations and rotations [2].
Revised: Additionally, there is work developing network-/kernel-representations which are invariant to permutations and rotations [2].

Original: Linear mode connectivity has also been investigated in different contexts, e.g. focusing on linear connections between different points from a training trajectory of neural networks [13]; focussing on linear connections between networks trained on independent data samples from the same initialisation [28]; relating linear mode connectivity to flatness and generalisation [15], and relating it to sparsity and the lottery ticket hypothesis [9, 10].
Revised: Linear mode connectivity has also been investigated in different contexts, e.g. focusing on linear connections between different points from a training trajectory of neural networks [13]; focussing on linear connections between networks trained on independent data samples from the same initialisation [28]; relating linear mode connectivity to flatness and generalisation [15], and relating it to sparsity and the lottery ticket hypothesis [9, 10].

Original: Appendix B. Finding good permutations at initialisation without training
Revised: Appendix B. Finding good permutations at initialisation without training

Original: In the main paper we described how a permutation obtained by training networks and matching their features leads to averaged initialisations that perform above chance.
Revised: In the main paper we described how a permutation obtained by training networks and matching their features leads to averaged initialisations that perform above chance.

Original: Here, we briefly note that such permutations can also be obtained without training the networks and describe how this can be done.
Revised: Here, we briefly note that such permutations can also be obtained without training the networks and describe how this can be done.

Original: As a first observation note that for our desired permutation π , the loss decreases when we start at θ 2 and move towards π ( θ 1 ) , i.e. in the direction of v π = π ( θ 1 ) − θ 2 .
Revised: As a first observation note that for our desired permutation π , the loss decreases when we start at θ 2 and move towards π ( θ 1 ) , i.e. in the direction of v π = π ( θ 1 ) − θ 2 .

Original: This suggests that the direction v π should be (negatively) aligned with the gradient at θ 2 .
Revised: This suggests that the direction v π should be (negatively) aligned with the gradient at θ 2 .

Original: Thus, we can try to find permutations that minimize the dot product v π · ∂L ( θ )
Revised: Thus, we can try to find permutations that minimize the dot product v π · ∂L ( θ )

Original: ∂θ | θ = θ 2 .
Revised: ∂θ | θ = θ 2 .

Original: This can be formalised as a minimum matching problem again and thus be solved with the Hungarian algorithm.
Revised: This can be formalised as a minimum matching problem again and thus be solved with the Hungarian algorithm.

Original: The idea described above does not yet give good permutations and we need two additional ideas.
Revised: The idea described above does not yet give good permutations and we need two additional ideas.

Original: • First, and most importantly, we replace the vanilla gradient ∂L ( θ ) ∂θ | θ = θ 2 by the update direction of a more powerful optimizer.
Revised: • First, and most importantly, we replace the vanilla gradient ∂L ( θ ) ∂θ | θ = θ 2 by the update direction of a more powerful optimizer.

Original: We used FOOF [3], which was inspired by obtaining a better understanding of KFAC [6, 24].
Revised: We used FOOF [3], which was inspired by obtaining a better understanding of KFAC [6, 24].

Original: Second, we symmetrise the loss.
Revised: Second, we symmetrise the loss.

Original: I
Revised: I

Original: e. (using the vanilla gradient rather than more advanced optimizers for notational simplicity) we minimize
Revised: e. (using the vanilla gradient rather than more advanced optimizers for notational simplicity) we minimize

Original: [Equation]
Revised: [Equation]

Original: We note that this gives permutations with very similar performance (at initialisation) as the permutations described in the main text.
Revised: We note that this gives permutations with very similar performance (at initialisation) as the permutations described in the main text.

Original: Moreover, in the case of CNNs this algorithm also does not find permutations which put different initialisations in the same loss valley, further supporting our finding from the main paper that CNN initialisations live in distinct loss valleys.
Revised: Moreover, in the case of CNNs this algorithm also does not find permutations which put different initialisations in the same loss valley, further supporting our finding from the main paper that CNN initialisations live in distinct loss valleys.

Original: Appendix C. Finding permutations for second-order Taylor expansions around the loss
Revised: Appendix C. Finding permutations for second-order Taylor expansions around the loss

Original: The algorithm described in the preceding section (almost) amounts to finding a permutation that minimizes a first-order Taylor expansion of the loss around θ 2 evaluated at 12 π ( θ 1 ) + 12 θ 2 . 

Revised: The algorithm described in the preceding section (almost) amounts to finding a permutation that minimizes a first-order Taylor expansion of the loss around θ 2 evaluated at 12 π ( θ 1 ) + 12 θ 2 .

Original: Thus or otherwise motivated, one might be tempted to find permutations that minimize higher-order Taylor expansions of the loss around θ 2 evaluated at 12 θ 2 + 12 π ( θ 1 ) both for initialisations or trained networks θ 2 .
Revised: Thus or otherwise motivated, one might be tempted to find permutations that minimize higher-order Taylor expansions of the loss around θ 2 evaluated at 12 θ 2 + 12 π ( θ 1 ) both for initialisations or trained networks θ 2 .

Original: If one uses KFAC [24] to obtain a computationally tractable approximation of the second-order expansion of the loss around θ 2 , and then computes the permutations sequentially layer-by-layer, the minimization problem to find good permutations turns out to be equivalent to the quadratic assignment problem (and thus NP hard).
Revised: If one uses KFAC [24] to obtain a computationally tractable approximation of the second-order expansion of the loss around θ 2 , and then computes the permutations sequentially layer-by-layer, the minimization problem to find good permutations turns out to be equivalent to the quadratic assignment problem (and thus NP hard).

Original: We experimented with convex relaxations of this problem over the Birkhoff polytope but failed to find better permutations than with the simpler method described in the main paper.
Revised: We experimented with convex relaxations of this problem over the Birkhoff polytope but failed to find better permutations than with the simpler method described in the main paper.

Original: Appendix D. Variants of feature matching and interpolation
Revised: Appendix D. Variants of feature matching and interpolation

Original: We experimented with different similarity functions k ( · , · ) for measuring the similarity between neuron activations, in particular ℓ 2 -distance and cosine similarity.
Revised: We experimented with different similarity functions k ( · , · ) for measuring the similarity between neuron activations, in particular ℓ 2 -distance and cosine similarity.

Original: We also investigated extracting neuron activations both before and after the non-linearity.
Revised: We also investigated extracting neuron activations both before and after the non-linearity.

Original: Broadly speaking, all variants performed very similarly.
Revised: Broadly speaking, all variants performed very similarly.

Original: In addition, we experimented with combining the permutations with learned / nonlearned “small” rotations.
Revised: In addition, we experimented with combining the permutations with learned / nonlearned “small” rotations.

Original: This often decreased the loss barrier slightly, but did not affect results qualitatively.
Revised: This often decreased the loss barrier slightly, but did not affect results qualitatively.

Original: Moreover, we tried interpolating between each layers’ weights using a rotation, rather than a linear path, but found that this significantly increased the loss barrier.
Revised: Moreover, we tried interpolating between each layers’ weights using a rotation, rather than a linear path, but found that this significantly increased the loss barrier.

Original: Finally, we tried matching weights instead of neuron activations.
Revised: Finally, we tried matching weights instead of neuron activations.

Original: Concretely, rather than finding a permutation that maximizes similarity between activation vectors, we found a permutation that maximizes similarity between weight vectors.
Revised: Concretely, rather than finding a permutation that maximizes similarity between activation vectors, we found a permutation that maximizes similarity between weight vectors.

Original: In principle, for every layer except the first and the last, this requires computing two permutations.
Revised: In principle, for every layer except the first and the last, this requires computing two permutations.

Original: To circumvent this problem, we simply started with the first layer, and moved upwards step by step.
Revised: To circumvent this problem, we simply started with the first layer, and moved upwards step by step.

Original: Matching weights performed slightly worse than matching activations, i.e. the loss barrier increased.
Revised: Matching weights performed slightly worse than matching activations, i.e. the loss barrier increased.

Original: Appendix E. Recap of permutation invariance
Revised: Appendix E. Recap of permutation invariance

Original: It is well known that neural networks are permutation invariant.
Revised: It is well known that neural networks are permutation invariant.

Original: To make this term precise, consider a fully connected network consisting of a sequence of weight matrices W 1 , . .
Revised: To make this term precise, consider a fully connected network consisting of a sequence of weight matrices W 1 , . .

Original: . , W ℓ , where W i has dimensions n i − 1 × n i , and a pointwise non-linearity σ (for simplicity of notation, we assume that each layer is followed by the same non-linearity, but this assumption can easily be dropped).
Revised: . , W ℓ , where W i has dimensions n i − 1 × n i , and a pointwise non-linearity σ (for simplicity of notation, we assume that each layer is followed by the same non-linearity, but this assumption can easily be dropped).

Original: For a given input X , the output of the i -th layer
Revised: For a given input X , the output of the i -th layer

Original: A i is defined iteratively as
Revised: A i is defined iteratively as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 6) and the output of the neural network is simply the output of the last layer A ℓ .
Revised: and the output of the neural network is simply the output of the last layer A ℓ .

Original: If we are given a sequence of permutation matrices P 0 , . .
Revised: If we are given a sequence of permutation matrices P 0 , . .

Original: P ℓ , where P i has dimensions n i × n i and where the first and the last permutation matrix P 0 , P ℓ are the identity, then the permuted neural network consisting of the weight matrices
Revised: P ℓ , where P i has dimensions n i × n i and where the first and the last permutation matrix P 0 , P ℓ are the identity, then the permuted neural network consisting of the weight matrices

Original: [Equation]
Revised: [Equation]

Original: computes precisely the same output as the network consisting of weight matrices
Revised: computes precisely the same output as the network consisting of weight matrices

Original: W i (since nonlinearity and permutation commute and since P Ti P i = Id ).
Revised: W i (since nonlinearity and permutation commute and since P Ti P i = Id ).

Original: Thus the input-output function specified by the network is invariant to permutations as described above.
Revised: Thus the input-output function specified by the network is invariant to permutations as described above.

Original: While we used fully connected networks above, permutation invariance straightforwardly applies for example to convolutional networks, where channels can be permuted.
Revised: While we used fully connected networks above, permutation invariance straightforwardly applies for example to convolutional networks, where channels can be permuted.

Original: Appendix F. Experimental details
Revised: Appendix F. Experimental details

Original: The term “SGD” in the paper usually refers to SGD with the usual 0.9 heavy-ball momentum, but we obtained qualitatively similar results for SGD without momentum.
Revised: The term “SGD” in the paper usually refers to SGD with the usual 0.9 heavy-ball momentum, but we obtained qualitatively similar results for SGD without momentum.

Original: We used a batch size of in all experiments, but we found similar results across a range of batch sizes.
Revised: We used a batch size ofin all experiments, but we found similar results across a range of batch sizes.

Original: The permutations are usually computed using the activations of 10000 images, and we verified that the quality of permutations saturates already for smaller numbers of images.
Revised: The permutations are usually computed using the activations of 10000 images, and we verified that the quality of permutations saturates already for smaller numbers of images.

Original: We used Kaiming-He init and ReLU nonlinearities for intermediate layers.
Revised: We used Kaiming-He init and ReLU nonlinearities for intermediate layers.

Original: The final layer is followed by a softmax, in combination with the cross-entropy loss.
Revised: The final layer is followed by a softmax, in combination with the cross-entropy loss.

Original: F.1. Fully connected networks
Revised: F.1. Fully connected networks

Original: Networks presented here are trained for 5 epochs with AdamW PyTorch [29] default settings, except Figure 3 mid-panel, where permutations were obtained by training for one epoch.
Revised: Networks presented here are trained for 5 epochs with AdamW PyTorch [29] default settings, except Figure 3 mid-panel, where permutations were obtained by training for one epoch.

Original: We note that for narrow networks, permutations that obtain even better performance when used to average initialisations can be obtained by training shorter.
Revised: We note that for narrow networks, permutations that obtain even better performance when used to average initialisations can be obtained by training shorter.

Original: F.2. CNNs
Revised: F.2. CNNs

Original: All CNN architectures use a sequence of convolutional filters and 2x2 average pooling layers, based on standard CNN design practices.
Revised: All CNN architectures use a sequence of convolutional filters and 2x2 average pooling layers, based on standard CNN design practices.

Original: These layers are finally followed by a global average pooling layer and one fully connected layer.
Revised: These layers are finally followed by a global average pooling layer and one fully connected layer.

Original: Based on this general design, we trained two kinds of CNN architectures, one of type
Revised: Based on this general design, we trained two kinds of CNN architectures, one of type

Original: [Equation]
Revised: [Equation]

Original: where “avg” denotes a 2x2 average pooling layer and the remaining numbers denote the number of (hidden) channels in the convolutional layers; and another one of type:
Revised: where “avg” denotes a 2x2 average pooling layer and the remaining numbers denote the number of (hidden) channels in the convolutional layers; and another one of type:

Original: [Equation]
Revised: [Equation]

Original: We used k = 32 , 64 , 128 ,
Revised: We used k = 32 , 64 , 128 ,

Original: The CNN in Figure 4 is of type 128 − 128 − avg − 256 − 256 , but we observed similar behaviour across different optimizers, architectures and datasets.
Revised: The CNN in Figure 4 is of type 128 − 128 − avg − 256 − 256 , but we observed similar behaviour across different optimizers, architectures and datasets.

Original: One more example for CIFAR10, AdamW and type 64 − 64 − avg − 128 − 128 is shown in Figure 10.
Revised: One more example for CIFAR10, AdamW and type 64 − 64 − avg − 128 − 128 is shown in Figure 10.

Original: Appendix G. Additional figures
Revised: Appendix G. Additional figures

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Original: [Figure]
Revised: [Figure]

Original: a i n
Revised: a i n

Original: N LL
Revised: N LL

Original: [Figure]
Revised: [Figure]

Original: Figure
Revised: Figure

Original: Loss increases at very high width, because in this case lower learning rates would be optimal, but we stick to AdamW default settings.
Revised: Loss increases at very high width, because in this case lower learning rates would be optimal, but we stick to AdamW default settings.

Original: [Figure]
Revised: [Figure]

Original: r a i n
Revised: r a i n

Original: N LL
Revised: N LL

Original: [Figure]
Revised: [Figure]

Original: Loss increases at very high width, because in this case lower learning rates would be optimal, but we stick to AdamW default settings.
Revised: Loss increases at very high width, because in this case lower learning rates would be optimal, but we stick to AdamW default settings.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]