Original: LieGG: Studying Learned Lie Group Generators
Revised: LieGG: Studying Learned Lie Group Generators

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Symmetries built into the network have appeared to be very beneficial for a widerange of tasks as it saves data to learn them.
Revised: Symmetries built into the network have appeared to be very beneficial for a widerange of tasks as it saves data to learn them.

Original: We depart from the position thatwhenthe symmetries are not built into a model a priori, it is advantageous for robustnetworks to  learnthe symmetriesdirectly from the data  to fit the task function.
Revised: Even when the exact type of symmetry is not known from prior knowledge, we depart from the position that it isadvantageous for robust networks to first learn symmetries from the data (if any)rather than starting to learn from no symmetry at all.

Original: In this paper, we present a method to extract symmetries learned by the neuralnetwork and to evaluate the degree to which the network is invariant to them.
Revised: In this paper, we present amethod to extract symmetries learned by the neural network and to evaluate thedegree to which the network is invariant to them.

Original: With our method, we are able to explicitly retrieve learned invariances in a formof the generators of corresponding Lie-groups without prior knowledge of thesymmetries in the data.
Revised: With our method, we are able toexplicitly retrieve learned invariances in a form of the generators of corresponding Lie-groups without prior knowledge of the symmetries in the data.

Original: We use the proposed method to study how symmetricalproperties depend on the network’s parameterization and configuration.
Revised: We use theproposed method to study how symmetrical properties depend on the network’sparameterization and configuration.

Original: We foundthat the ability of the network to learn symmetries generalizes over a wide rangeof architectures.
Revised: We found that the ability of the network tolearn symmetries generalizes over a wide range of architectures.

Original: However, the quality of the learned symmetries depends on thedepth and number of parameters.
Revised: However, thequality of the learned symmetries depends on the depth and number of parameters.

Original:  Introduction
Revised:  Introduction

Original: Convolutional Neural Networks (CNNs) are efficient, for one because they can convert translationsymmetry in the data into a built-in translation-equivariance property of the network without exhausting data to learn the equivariance.
Revised: Convolutional Neural Networks (CNNs) are efficient, for one because they can convert translationsymmetry in the data into a built-in translation-equivariance property of the network without exhausting data to learn the equivariance.

Original: Group-equivariant networks generalize this property to rotation[1, 2, 3, 4], scale [5, 6, 7], and other symmetries defined by matrix groups [8].
Revised: Group-equivariant networks generalize this property to rotation[1, 2, 3, 4], scale [5, 6, 7] and other symmetries defined by matrix groups [8].

Original: Equipping a neuralnetwork with prior known symmetries has proved to be data-efficient.
Revised: Equipping a neuralnetwork with prior known symmetries has proved to be data-efficient.

Original: Recent works [9, 10, 11] have demonstrated that hard coding the symmetry into the model does notalways lead to a better generalization.
Revised: Recent works [9, 10, 11] have demonstrated that hard coding the symmetry into the model does notalways lead to a better generalization.

Original: Often, soft or learned equivariance is more advantageousmostly in terms of data efficiency and accuracy [12].
Revised: Often, soft or learned equivariance is more advantageousmostly in terms of data efficiency and accuracy [12].

Original: Moreover, architectures with no equivariancebuilt-in, such as transformers [13] and multi-layer perceptron mixers [14], achieve a remarkableperformance on a wide range of problems.
Revised: Moreover, architectures with no equivariancebuilt-in, such as transformers [13] and multi-layer perceptron mixers [14], achieve a remarkableperformance on a wide range of problems.

Original: Effectively, they learn their own geometrical priors withoutexplicit symmetry constraints [15].
Revised: Effectively, they learn their own geometrical priors withoutexplicit symmetry constraints [15].

Original: This raises the following questions to study in this paper.
Revised: This raises the following questions to study in this paper.

Original: To whatdegree do neural networks learn symmetries?
Revised: To whatdegree do neural networks learn symmetries?

Original: How accurately do learned symmetries reflect the truesymmetries in the data?
Revised: How accurately do learned symmetries reflect the truesymmetries in the data?

Original: Can we support the capability of networks for learning symmetries?
Revised: Can we support the capability of networks for learning symmetries?

Original: Wepresent a method to study symmetries learned by networks to address the questions.
Revised: Wepresent a method to study symmetries learned by networks to address the questions.

Original: In the paper we depart from Lie group theory and develop a method that can retrieve symmetrieslearned from the data by any neural network.
Revised: In the paper we depart from Lie group theory and develop a method that can retrieve symmetrieslearned from the data by any neural network.

Original: Our method makes the assumption that the model isdifferentiable, commonly met in neural networks.
Revised: Our method makes the assumption that the model isdifferentiable, commonly met in neural networks.

Original: While previous works on analyzing symmetries innetworks rely on empirical analysis of network representations [16] or on examination of a givenset of transformations [17, 6, 18], our method outputs a generator of the corresponding Lie-group toquantitatively evaluate how sensitive the network is in the direction of the learned symmetry (Figure1).
Revised: While previous works on analyzing symmetries innetworks rely on empirical analysis of network representations [16] or on examination of a givenset of transformations [17, 6, 18], our method outputs a generator of the corresponding Lie-group toquantitatively evaluate how sensitive the network is in the direction of the learned symmetry (Figure1).

Original: We make the following contributions:
Revised: We make the following contributions:

Original: • From the perspective of Lie-groups, we propose a theory to study symmetrical properties of
Revised: • From the perspective of Lie-groups, we propose a theory to study symmetrical properties of

Original: a trained neural network.
Revised: a trained neural network.

Original: • We derive an efficient implementation that allows improving the interpretability of the modelby revealing symmetries it learned.• With our method we conclude that models with more parameters and gradual fine-tuninglearn more precise symmetry groups with a higher degree of invariances to them.
Revised: • We derive an efficient implementation that allows improving the interpretability of the modelby revealing symmetries it learned.• With our method we conclude that models with more parameters and gradual fine-tuninglearn more precise symmetry groups with a higher degree of invariances to them.

Original:  Related work
Revised:  Related work

Original: Equivariantand invariant networks The goal of equivariantand invariant networks is to buildthe symmetries into a network architecture as an inductive bias.
Revised: Equivariant networks The goal of equivariant networks is to build the symmetries into a networkarchitecture as an inductive bias.

Original: Starting from the convolutionalnetworks [19] that contain translation symmetry, the concept of equivariance was generalized torotations [1, 2, 3, 4], permutations [20], scaling [5, 6,21, 7, 22] and arbitrary matrix groups in [8].
Revised: Starting from the convolutional networks [19] that contain translationsymmetry, the concept of equivariant networks was generalized to rotations [1, 2, 3, 4], permutations[20], scaling [5, 6, 7, 21] and arbitrary matrix groups in [8].

Original: Various methods have been proposed for learning symmetries directly from the training data [11, 23]among which a popular approach is  to learn transformations by estimating the infinitesimal generators[24, 25, 26, 27, 28].
Revised: Recent works also include [11, 22],where the network is designed to be equivariant to symmetries learned directly from the data.

Original: These papers focus on answering the question of how to build symmetries intothe models by modifying the  architecture.
Revised: Thesepapers focus on answering the question of how to build symmetries into the network by modifyingthe architecture.

Original: In our work, we focus on the reverse question, i.e. given anetwork with a fixed architecture, what symmetries does it learn from the data.
Revised: In our work, we focus on the reverse question, i.e. given a network with a fixedarchitecture, what symmetries does it learn from the data.

Original: Symmetries in neural networks Another line of work is focused on interpreting symmetriesin neural networks.
Revised: Symmetries in neural networks Another line of work is focused on interpreting symmetriesin neural networks.

Original: In [16] Olah et al.
Revised: In [16] Olah et al.

Original: take inspiration from biological circuit motifs [29] andstudy the equivariant patterns learned by a neural network on the image classification task.
Revised: take inspiration from biological circuit motifs [23] andstudy the equivariant patterns learned by a neural network on the image classification task.

Original: Theauthors demonstrate that a network learns rotation, hue, and scale symmetries when trained on
Revised: Theauthors demonstrate that a network learns rotation, hue, and scale symmetries when trained on

Original: ImageNet.
Revised: ImageNet.

Original: In [17] Goodfellow et al.
Revised: In [17] Goodfellow et al.

Original: propose a number of empirical tests to study invariancesto known transformations in the network, and demonstrate that auto-encoding architectures learnincreasingly invariant features in deep layers of the network.
Revised: propose a number of empirical tests to study invariancesto known transformations in the network, and demonstrate that auto-encoding architectures learnincreasingly invariant features in deep layers of the network.

Original: Along the same lines, Lenc & Vedaldi[18] quantify invariances and equivariance in the layers of convolutional networks to a pre-definedset of transformations.
Revised: Along the same lines, Lenc & Vedaldi[18] quantify invariances and equivariance in the layers of convolutional networks to a pre-definedset of transformations.

Original: In this paper, we also propose a method to study the symmetrical propertiesof neural networks.
Revised: In this paper, we also propose a method to study the symmetrical propertiesof neural networks.

Original: In contrast to existing works, our method does not require knowing the setof transformations beforehand, and it also provides theoretical guarantees of invariances from theperspective of the Lie groups theory.
Revised: In contrast to existing works, our method does not require knowing the setof transformations beforehand, and it also provides theoretical guarantees of invariances from theperspective of the Lie groups theory.

Original: Network configuration: width, depth, number of parameters Neural networks of various widthsand depths have been studied through the lens of universal approximation theorem [30, 31], functionalexpressiveness [32] and by empirical analysis of learned representations [33].
Revised: Network configuration: width, depth, number of parameters Neural networks of various widthsand depths have been studied through the lens of universal approximation theorem [24, 25], functionalexpressiveness [26] and by empirical analysis of learned representations [27].

Original: These works focuson either characterizing the learning ability of the model or on interpreting the discrepancy betweenrepresentations that models with different architectures learn.
Revised: These works focuson either characterizing the learning ability of the model or on interpreting the discrepancy betweenrepresentations that models with different architectures learn.

Original: They, however, do not analyze how thesymmetrical properties of a model depend on its width and depth.
Revised: They, however, do not analyze how thesymmetrical properties of a model depend on its width and depth.

Original: Other works focus on analyzing the connection between a number of parameters and the generalization capability of networks [34, 35, 36, 37].
Revised: Other works focus on analyzing the connection between a number of parameters and the generalization capability of networks [28, 29, 30, 31].

Original: It is commonly argued that overparametrizedarchitectures achieve better generalization bounds, i.e. provide smaller discrepancy between train andtest performance and are thus preferred in many applications [37].
Revised: It is commonly argued that overparametrizedarchitectures achieve better generalization bounds, i.e. provide smaller discrepancy between train andtest performance and are thus preferred in many applications [31].

Original: However, it is unclear if modelswith more parameters and hence with a better generalization capability also learn better symmetries.
Revised: However, it is unclear if models with more parameters and hence with a better generalization capability also learn better symmetries.

Original: In our paper, we investigate this for a family of feed-forward networks.
Revised: In our paper, we investigate this for a family of feed-forward networks.

Original: Robust Learning Various methods have been proposed to train robust models on small datasets.
Revised: Robust Learning Various methods have been proposed to train robust models on small datasets.

Original: While the naive training may lead to overfitting on the training subset [38], well-organized pretraininghelps to mitigate this issue.
Revised: While the naive training may lead to overfitting on the training subset [32], well-organized pretraininghelps to mitigate this issue.

Original: Features extracted by a model pretrained on a bigger dataset oftendemonstrate more robust results [39].
Revised: Features extracted by a model pretrained on a bigger dataset oftendemonstrate more robust results [33].

Original: While there are many useful techniques for training effectivefeature extractors [40, 41, 42], the analysis of such methods from the invariance point of view wasnot done before.
Revised: While there are many useful techniques for training effectivefeature extractors [34, 35, 36], the analysis of such methods from the invariance point of view wasnot done before.

Original: We demonstrate that our method allows for better understanding and explainingwhat training regimes lead to more invariant, and thus more robust features.
Revised: We demonstrate that our method allows for better understanding and explainingwhat training regimes lead to more invariant, and thus more robust features.

Original:  Background
Revised:  Background

Original: The focus of this paper is a symmetry group .
Revised: The focus of this paper is a symmetry group .

Original: Intuitively, a symmetry is a transformation of an objectthat leaves the object unchanged.
Revised: Intuitively, a symmetry is a transformation of an objectthat leaves the object unchanged.

Original: The set of all such transformations of an object with compositionas a binary operation forms a group.
Revised: The set of all such transformations of an object with compositionas a binary operation forms a group.

Original: In this paper, the object of interest is a dataset, and we studywhat kind of transformations map one data sample to another and how neural networks preserveinformation about symmetries.
Revised: In this paper, the object of interest is a dataset, and we studywhat kind of transformations map one data sample to another and how neural networks preserveinformation about symmetries.

Original: The theory of Lie groups is a sweet spot of mathematics that helps to formalize symmetries andprovides practical methods for studying them.
Revised: The theory of Lie groups is a sweet spot of mathematics that helps to formalize symmetries andprovides practical methods for studying them.

Original: Formally, a Lie group is a group that has the structureof a differential manifold.
Revised: Formally, a Lie group is a group that has the structureof a differential manifold.

Original: The tangent space in the identity element of a Lie group forms a vectorspace called Lie algebra.
Revised: The tangent space in the identity element of a Lie group forms a vectorspace called Lie algebra.

Original: A Lie algebra determines a Lie group up to an isomorphism for a simplyconnected Lie groups and, being a vector space, is a more convenient object to study than a Lie group.
Revised: A Lie algebra determines a Lie group up to an isomorphism for a simplyconnected Lie groups and, being a vector space, is a more convenient object to study than a Lie group.

Original: Thus, in order to recover asimply connected symmetry group, it is sufficient to understand its Liealgebra.
Revised: Thus, in order to recover a symmetry group, it is sufficient to understand its Lie algebra.

Original: In this paper, we assume that a Lie group G is a matrix Lie group, i.e., a closed subgroup of thegeneral linear group degree n , denoted as GL ( n ) and is defined as the set of n × n invertible matrices.
Revised: In this paper, we assume that a Lie group G is a matrix Lie group, i.e., a closed subgroup of thegeneral linear group degree n , denoted as GL ( n ) and is defined as the set of n × n invertible matrices.

Original: The correspondence between the Lie group G and its Lie algebra g in this case is given by theexponential map:
Revised: The correspondence between the Lie group G and its Lie algebra g in this case is given by theexponential map:

Original: [Equation]
Revised: [Equation]

Original: where M ( n ) is the set of all n × n matrices.
Revised: where M ( n ) is the set of all n × n matrices.

Original: Each such element h from Equation 1 is called aninfinitesimal generator of the Lie group G .
Revised: Each such element h from Equation 1 is called aninfinitesimal generator of the Lie group G .

Original: We introduce LieGG , a method which extracts Lie G roup G enerators, i.e. finds the infinitesimalgenerators of the Lie algebra.
Revised: We introduce LieGG , a method which extracts Lie G roup G enerators, i.e. finds the infinitesimalgenerators of the Lie algebra.

Original: To find the Lie algebra basis, we use the discriminator of the dataset,i.e. a function f : R n → R such that f ( x ) = 

Revised: To find the Lie algebra basis, we use the discriminator of the dataset,i.e. a function f : R n → R such that f ( x ) =

Original: if and only if x is an element of the dataset D data.
Revised: if and only if x is an element of the dataset D data.

Original: Having a discriminator, we use the following criterion for a symmetry group of algebraic equations:
Revised: Having a discriminator, we use a well-known criterion for a symmetry group of algebraic equations:

Original: Theorem 3.1 (Theorem 2.8 [43]) .
Revised: Theorem 3.1 (Theorem 2.8 [37]) .

Original: Let G be a connected Lie group of linear transformations actingon an n -th dimensional manifold X .
Revised: Let G be a connected Lie group of linear transformations actingon an n -th dimensional manifold X .

Original: Let F : X → R l , l ≤ n , define a system of algebraic equations:
Revised: Let F : X → R l , l ≤ n , define a system of algebraic equations

Original: [Equation]
Revised: [Equation]

Original: and assume that the system is of maximal rank, meaning that the Jacobian matrix (cid:16) [Equation] ∂x k (cid:17) is of rank lfor every solution x of the system.
Revised: and assume that the system is of maximal rank, meaning that the Jacobian matrix (cid:16) ∂F ν ∂x k (cid:17) is of rank lfor every solution x of the system.

Original: Then G is a symmetry group of the system if and only if
Revised: Then G is a symmetry group of the system if and only if

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for every infinitesimal generator h of G , where h ij is an element of the matrix h in the i -th row andj -th column.
Revised: for every infinitesimal generator h of G , where h ij is an element of the matrix h in the i -th row andj -th column.

Original: Example Let us illustrate the practical significance of the theorem with an example.
Revised: Example Let us illustrate the practical significance of the theorem with an example.

Original: Suppose data
Revised: Suppose data

Original: D lie on a sphere in R n :
Revised: D lie on a sphere in R n :

Original: [Equation]
Revised: [Equation]

Original: The discriminator for this example is the function f = x 21 + · · · + x 2 n − 1 .
Revised: The discriminator for this example is the function f = x 2 1 + · · · + x 2 n − 1 .

Original: According to the theorem,in order to find Lie algebra of the symmetry group, we can find a basis of the solution of the followinglinear equations, where a ij are sought variables:
Revised: According to the theorem,in order to find Lie algebra of the symmetry group, we can find a basis of the solution of the followinglinear equations, where a ij are sought variables:(cid:88) [Equation] j

Original: [Equation]
Revised: [Equation]

Original: when x 21 + · · · + x 2 n = 1 .
Revised: when x 21 + · · · + x 2 n = 1 .

Original: We assert, that the solutions are matrices
Revised: We assert, that the solutions are matrices

Original: A such that A
Revised: A such that A

Original: +
Revised: +

Original: A
Revised: A

Original: T
Revised: T

Original: = 0 .
Revised: = 0 .

Original: Indeed, for x : x i = 1 , x j = 0 , j ̸
Revised: Indeed, for x : x i = 1 , x j = 0 , j ̸

Original: = i , the condition means that a ii = 0 .
Revised: = i , the condition means that a ii = 0 .

Original: For x : x i = 1 √ 2 , x j =1 √ 2 , x k = 0 , k ̸
Revised: For x : x i = 1 √ 2 , x j =1 √ 2 , x k = 0 , k ̸

Original: = i, j , it follows, that a ij + a ji = 0 .
Revised: = i, j , it follows, that a ij + a ji = 0 .

Original: On the other hand if A = − A T , then(cid:80) i ̸
Revised: On the other hand if A = − A T , then(cid:80) i ̸

Original: = j (cid:0) x i · a ij · x j + x j · a ji · x i (cid:1)
Revised: = j (cid:0) x i · a ij · x j + x j · a ji · x i (cid:1)

Original: = (cid:80) i ̸
Revised: = (cid:80) i ̸

Original: =
Revised: =

Original: j (cid:0) x i · a ij · x j − x i · a ij · x j (cid:1) = 0 .
Revised: j (cid:0) x i · a ij · x j − x i · a ij · x j (cid:1) = 0 .

Original: Note that the Lie algebra of the matrices{ A
Revised: Note that the Lie algebra of the matrices{ A

Original: [Equation]
Revised: [Equation]

Original: A + A T = 0 } corresponds to the Lie group of matrices
Revised: A + A T = 0 } corresponds to the Lie group of matrices

Original: [Equation]
Revised: [Equation]

Original: which is a rotation group.
Revised: which is a rotation group.

Original: Thus the symmetry of a sphere is a rotation group.
Revised: Thus the symmetry of a sphere is a rotation group.

Original:  Method
Revised:  Method

Original: To compute a Lie algebra given a discriminator function f : R n → R we use Theorem 3.1 and solvethe system of linear equations, where each equation corresponds to one element in the dataset:
Revised: To compute a Lie algebra given a discriminator function f : R n → R we use theorem 3.1 and solvethe system of linear equations, where each equation corresponds to one element in the dataset:

Original: [Equation]
Revised: [Equation]

Original:  each element from the dataset.
Revised: for each element from the dataset.

Original: [Equation]
Revised: [Equation]

Original: This is a system of linear equations with n 2 variables, and number of equations equal to the numberof points in the data.
Revised: This is a system of linear equations with n 2 variables, and number of equations equal to the number ofpoints in the data.

Original: We denote the matrix defined by equations 2 as E and call it the networkpolarization matrix .
Revised: We denote the matrix defined by equations 2 as E and call it network polarizationmatrix .

Original: The problem of finding a basis of solutions of linear equations is a standardproblem in numerical analysis, that can be solved using singular value decomposition (SVD).
Revised: The problem of finding a basis of solutions of linear equations is a standard problem innumerical analysis, that can be solved using singular value decomposition (SVD).

Original: Weremind that we can write E using SVD in the following way:
Revised: We remind that wecan write E using SVD in the following way:

Original: [Equation]
Revised: [Equation]

Original: where U, V - are orthogonal matrixes and Σ is a diagonal matrix with decreasing elements on thediagonal.
Revised: where U, V - are orthogonal matrixes and Σ is a diagonal matrix with decreasing elements on thediagonal.

Original: Thus, the columns in V corresponding to almost 0 singular values form a Lie algebra basis.
Revised: Thus, the columns in V corresponding to almost 0 singular values form a Lie algebra basis.

Original: Thereby, in practice, the calculation of LieGG consists of 3 steps: 1.training a neural network;2.calculation of the polarization matrix E ; and 3.calculation of singular vectors corresponding toalmost zero singular values of the matrix E .
Revised: Thereby, in practice, the calculation of LieGG consists of 3 steps: 1.training a neural network;2.calculation of the polarization matrix E ; and 3.calculation of singular vectors corresponding toalmost zero singular values of the matrix E .

Original: 4.1(revised) Computation of Lie algebra of the group acting on R 2 .
Revised: 4.1 Computation of Lie algebra of the group acting on R 2 .

Original: Symmetries play an important role in computer vision problems, and we will describe LieGG for this case in more detail.
Revised: Symmetries play an important role in computer vision problems, and we will describe LieGG forthis case in more detail.

Original: In various computer vision problems, it is assumed that the symmetry group
Revised: In various computer vision problems, it is assumed that the symmetry group

Original: G changes images from the dataset by acting on R 2 .
Revised: G changes images from the dataset by acting on R 2 .

Original: For example, convolutional networks [19] and recently proposed equivariant networks for rotations [3] assume that the group acts as a translation or rotation of the image.
Revised: For example, convolutional networks [19] andrecently proposed equivariant networks for rotations [3] assume that the group acts as a translationor rotation of the image.

Original: These neural networks perceive each image as a function f with non-zero values on some compact subset Ω ⊂ R 2 , where the elements of the subset correspond to spatial coordinates. Thus, the space of images X is the space of all such functions: [Equation]
Revised: These neural networks perceive each image as a function f with non-zerovalues on some compact subset Ω ⊂ R 2 .

Original: A  vision model F : X → R is a function on such a space of functions.
Revised: A computer vision model F : f → R is a function on suchfunctions.

Original: We view an image is afunction on a pixel grid ( x 1 , · · · , x n ) of the domain Ω , i.e. theimage can be identified with thepoint in R n , which encode the pixel values. In this case the model takes as input n real values F ( f ( x 1 ) , · · · , f ( x n )) . 
Revised: Usually, this model takes as input the values of f in some grid ( x 1 , · · · , x n ) of the domainΩ , i.e. the model takes as input n real values F ( f ( x 1 ) , · · · , f ( x n )) . The most common example ofsuch a grid is pixel coordinates.

Original: It is important to note that Theorem 3.1 assumes that the group G acts continuously on the datamanifold X .
Revised: In the paper we assume that the group G acts continuously on the model F , where the group action isdefined as follows:

Original:  However, this assumption is violated for image data as an image  is a non-continuousfunction of coordinates due to the discrete pixel grid.
Revised: [Equation] However, since an image itself is a highly non continuous function out of coordinates, G does not actcontinuously on the computer vision model F .

Original: We overcome the discontinuity issue bypreprocessing images with Gaussian smoothing.
Revised: To overcome this problem, we preprocess images bydoing Gaussian smoothing.

Original: To use LieGG for image data, we explicitly write the condition that the matrix h is an infinitesimalgenerator for the image data X on the grid ( x 1 , · · · , x n ) , where x p [ j ] corresponds to the projectionof x p onto the j-th coordinate axis.
Revised: To use LieGG for image data, we rewrite the condition from the theorem 3.1 in the following way: [Equation]

Original: The condition reads as follows:
Revised: i [Equation]

Original: F
Revised: p

Original: [Equation]
Revised: [Equation]

Original: p = n,k =2 ,j =2 (cid:88) p =1 ,k =1 ,j = ∂F ∂f ( x p ) · ∂f ( x p ) ∂ x p [ k ] · h k,j · x p [ j ] = 0 (3) where t ∈ R .
Revised: ∂F ∂v

Original: The resulting condition is the equation in 4 variables h k,j , k
Revised: [Equation] ∂f [Equation] ∂y i

Original: = 1 , 2; j = 1 , 2 .
Revised: [Equation] [Equation]

Original: Thus, in this case, the polarization matrix has 4 columns. Thus, in equation 3 a space of potentialimage symmetries is reduced to 4-dimensional spatial subgroup.
Revised: (3) where t ∈ R , h from the algebra Lie, y i ( x j ) is an i -th coordinate of the data sample x j . The resultingequations are equations in 4 variables

Original: This is to provide a prior for the
Revised: h i,j , i = 1 , 2; j = 1 , 2 .

Original: LieGG to retrieve meaningful symmetries from for images as the full-scale space of the potentialsymmetries is overwhelming for image data. This is a strong yet proven to useful prior for visionmodels [1, 44, 2, 3, 4].
Revised: Thus, in this case, the polarizationmatrix is a matrix having only 4 columns.

Original: In practice, the calculation of LieGG for  vision models consists of the following steps: (i) preprocessing images by doing Gaussian smoothing, (ii) training a neural network, (iii) computing polarizationmatrix using equation 3.
Revised: In practice, the calculation of LieGG for computer vision models consists of the following steps:1.preprocessing images by doing Gaussian smoothing; 2.training a neural network; 3.computingpolarization matrix using equation 3, where [Equation]

Original: 4.2 Symmetry bias and symmetry variance
Revised: 4.2 Symmetry bias and symmetry variance

Original: The scope of LieGG is not limited to the study of symmetry, but also to the evaluation of the degreeof neural network invariance.
Revised: The scope of LieGG is not limited to the study of symmetry, but also to the evaluation of the degreeof neural network invariance.

Original: If we know the true symmetries, we can apply our method to analyzehow close the extracted symmetries are to the true ones; and the degree of network invariance to thesesymmetries.
Revised: If we know the true symmetries, we can apply our method to analyzehow close the extracted symmetries are to the true ones; and the degree of network invariance to thesesymmetries.

Original: The former we alias symmetry bias , and the later - symmetry variance .
Revised: The former we alias symmetry bias , and the later - symmetry variance .

Original: To understandthis we show how we can estimate the degree of network invariance to the element g of a Lie group:
Revised: To understandthis we show how we can estimate the degree of network invariance to the element g of a Lie group:

Original: [Equation]
Revised: [Equation]

Revised: x i ))

Revised: [Equation]

Revised: := ˜

Original:  Let h is an element from Lie algebra, that corresponds to g , i. [Equation]
Revised: R ( g ) . Let h is an element from Lie algebra, that corresponds to g , i.e.

Original: [Equation]
Revised: [Equation]

Original: We denote the matrix with the i -th row equal to ∂F ( x i ) ∂x k · x ij as E .
Revised: We denote the matrix with the i -th row equal to ∂F ( x i ) ∂x k · x ij as E .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where h is a vector with coordinates equal to h kj .
Revised: where h is a vector with coordinates equal to h kj .

Original: To calculate the value of E · h we use SVD ofmatrix E .
Revised: To estimate the value of E · h we use SVD ofmatrix E .

Original: Lets E = U Σ V ∗ , where U, V - are semi-unitary matrixes and Σ is a diagonal matrix, andh = (cid:80) l h l v i .
Revised: Lets E = U Σ V ∗ , where U, V - are semi-unitary matrixes and Σ is a diagonal matrix, andh = (cid:80) l h l v i .

Original: We can estimate ˜ R ( g ) in the following way:
Revised: We can estimate ˜ R ( g ) in the following way:

Original: [Equation]
Revised: [Equation]

Original: where σ l is the l -th singular value.
Revised: where σ l is the l -th singular value.

Original: In particular, the equation 4 shows that singular values reflect the degree of network invariance totransformations corresponding to singular vectors.
Revised: In particular, the equation 4 shows that singular values reflect the degree of network invariance totransformations corresponding to singular vectors.

Original: With this, we  use the smallest singular value ofthe network polarization matrix to define the symmetry variance as V ( E ) =
Revised: Thus, we can use the absolute values of the lastsingular values as an estimation of symmetry variance.

Original: |E| − 1 σ min ( E ) 2 , where|E| denotes number of rows in the network polarization matrix.

Original: Moreover, when the symmetries areknown, the difference between the singular vectors and thegenerator of the known symmetry groupequates to the symmetry bias , i.e. B i ( E , G ) =
Revised: Moreover, when the symmetries are known,the difference between the singular vectors and the known symmetries is a good estimation of thesymmetry bias.

Original: ||

Original: ˜ v i ( E ) − ˜ G|| F , where ˜ v i ( E ) returns the i-th singularvector reshaped to a matrix, and ˜ G is a ground truth symmetry group generator, which is the closestto ˜ v i ( E ) , e.g. the closest skew-symmetric matrix for O ( n ) .

Original:  Experiments
Revised:  Experiments

Revised: [Figure]

Original: 5.1 Implementation details
Revised: 5.1 Implementation details

Original:  We use a multi-layer perceptron with linear layers followed by Swish activation [45].
Revised: [Figure] We use a multi-layer perceptron with linear layers followed by Swish activation [38].

Original: For theregression experiment, we use the MLP with 3 layers and set the hidden dimension to 32 .
Revised: For theregression experiment, we use the MLP with 3 layers and set the hidden dimension to 32 .

Original: For theclassification experiment weuse the MLP with a classification head attached, and train variousconfigurations varying a number of parameters, depth, and width.
Revised: Forthe classification experiment we the MLP with a classification head attached, and train variousconfigurations varying a number of parameters, depth, and width.

Original: We train all of the models with
Revised: We train all of the models with

Original: Adam optimizer [46] setting the learning rate set to 10 − 3 .
Revised: Adam optimizer [39] setting the learning rate set to 10 − 3 .

Revised: To compute symmetry bias, we compute adistance with the group generator of interest, e.g. for a rotation group symmetry the bias equates tothe distance with a closes skew-symmetric matrix: B ( X )

Revised: =

Revised: ||

Revised: X − P ( X ) || F , where P is a projectiononto a set of skew-symmetric matrices.

Original: The results for all of the experiments areaveraged over 8 common fixed random seeds.
Revised: The results for all of the experiments are averaged over 8common fixed random seeds.

Original: 5.2 Retrieving symmetries from a synthetic regression
Revised: 5.2 Retrieving symmetries from a synthetic regression

Original: Firstly, we perform the experiment on a synthetic problem to test the ability of our method toaccurately retrieve the symmetry learned by the network.
Revised: Firstly, we perform the experiment on a synthetic problem to test the ability of our method toaccurately retrieve the symmetry learned by the network.

Original: To do so, we adopt the invariant regressiontask from [8] with clean O (5) symmetry built-in.
Revised: To do so, we adopt the invariant regressiontask from [8] with clean O (5) symmetry built-in.

Original: The regression function f :R 2 × 5 → R is given by:
Revised: The regression function f : 2 R 5 → R is given by:

Original: [Equation]
Revised: [Equation]

Original: With this, we construct the training dataset to fit the regression function with the multi-layer perceptron.
Revised: With this, we construct the training dataset to fit the regression function with the multi-layer perceptron.

Original: We note that the MLP is an unconstrained feed-forward neural network with no a priori knowledge ofsymmetries in the data.
Revised: We note that the MLP is an unconstrained feed-forward neural network with no a priori knowledge ofsymmetries in the data.

Original: Once the model is converged, we apply our method to analyze how close arethe retrieved symmetries to the clean O (5) group (symmetry bias), and how invariant is the networkto these symmetries (symmetry variance).
Revised: Once the model is converged, we apply our method to analyze how close arethe retrieved symmetries to the clean O (5) group (symmetry bias), and how invariant is the networkto these symmetries (symmetry variance).

Original: We also study how the ability of the network to learnsymmetries depends on a number of available training samples.
Revised: We also measure how the ability of the network to learnsymmetries depends on a number of available training samples.

Original: We report the results in Figure 2.
Revised: We report the results in Figure 2.

Original: As can be seen from Figure 2a, the spectrum of the network polarization matrix is zeroing, indicatingthat the model becomes invariant to the transformations encoded by the corresponding singularvectors, and these singular vectors are the generators of the rotation group as indicated by zero O (5)symmetry bias (Figure 2b).
Revised: As can be seen from Figure 2a, the spectrum of the network polarization matrix is zeroing, indicatingthat the model becomes invariant to the transformations encoded by the corresponding singularvectors, and these singular vectors are the generators of the rotation group as indicated by zero O (5)symmetry bias (Figure 2b).

Original: We also observed that the network can accurately learn the symmetrygroup generators even on a small training set size.
Revised: We also observed that the network can accurately learn the symmetrygroup generators even on a small training set size.

Original: However, a larger training set size results in thenetwork having smaller symmetry variance, i.e. being more invariant.
Revised: However, a larger training set size results in thenetwork having smaller symmetry variance, i.e. being more invariant.

Original: For additional validation, weused the LieGG to extract symmetries from the randomly initialized O (5) invariant EMLP [8].

Original: Forthe EMLP, the LieGG recorded zero symmetry variance and zero symmetry bias with respect to the

Original: O (5) group without any model pre-training.

Original: This experiment demonstrates that on a synthetic task, our method can accurately retrieve thesymmetries learned or built-in. When the symmetries are learned, the larger training dataset facilitateslearning invariances for the model.
Revised: This experiment demonstrates that on a synthetic task, our method can accurately retrieve thesymmetries learned by the network and that the larger training dataset facilitates learning invariancesfor the model.

Original: 5.3 Retrieving symmetries from rotation MNIST
Revised: 5.3 Retrieving symmetries from rotation MNIST

Original: In this experiment, we test the ability of our method to retrieve learned symmetries on a morechallenging rotation MNIST dataset [47].
Revised: In this experiment, we test the ability of our method to retrieve learned symmetries on a morechallenging rotation MNIST dataset [40].

Original: We train the feed-forward network to output the class of the digit invariant to 2D in-plane rotations.
Revised: We train the feed-forward network to output the class ofthe digit invariant to 2D in-plane rotations.

Original: The symmetry group that the model should learn is thus
Revised: The symmetry group that the model should learn is thus

Original: SO (2) .
Revised: SO (2) .

Original: We note that different from the synthetic dataset, the symmetries in the rotation MNIST aremuch noisier since the rotations are coarsened by the projection onto a pixel grid.
Revised: We note that different from the synthetic dataset, the symmetries in the rotation MNIST aremuch noisier since the rotations are coarsened by the projection onto a pixel grid.

Original: In addition, thenetwork has a classification error, hence it serves as only an approximate data discriminator function.
Revised: In addition, thenetwork has a classification error, hence it serves as only an approximate data discriminator function.

Original: In this scenario, we test how well our method can retrieve the noisy symmetries.
Revised: In this scenario, we test how well our method can retrieve the noisy symmetries.

Original: We experiment with two architectures: a shallow 2-layer perceptron and a deeper 6-layer networkwith 4 times number of parameters.
Revised: We experiment with two architectures: a shallow 2-layer perceptron and a deeper 6-layer networkwith 4 times number of parameters.

Original: Once, the networks’ performance plateaus on the validationsplit, we terminate the training, and apply our method to retrieve learned symmetries.
Revised: Once, the networks’ performance plateaus on the validationsplit, we terminate the training, and apply our method to retrieve learned symmetries.

Original: We plot thespectrum of the network polarization matrix, SO (2) symmetry bias and visualize learned symmetriesfor shallow and deep models in Figure 3.
Revised: We plot thespectrum of the network polarization matrix, SO (2) symmetry bias and visualize learned symmetriesfor shallow and deep models in Figure 3.

Original: From Figure 3a we observe that for both shallow and deep models, our method can robustly retrievethe rotation symmetry as indicates by zeroing symmetry bias with respect to SO (2) group, given the4-dimensional spatial subgroup prior.
Revised: From Figure 3a we observe that for both shallow and deep models, our method can robustly retrievethe rotation symmetry as indicates by zeroing symmetry bias with respect to SO (2) group.

Original: However, we noticed that the rotation learned by the shallowmodel is in general noisier than the symmetry learned by the deeper networks as indicated by a highersymmetry bias.
Revised: However,we noticed that the rotation learned by the shallow model is in general noisier than the symmetrylearn by the deeper networks as indicated by a higher symmetry bias.

Original: We visualize it in Figure 3b by sampling learned Lie algebras and applying them totransform an input image.
Revised: We visualize it in Figure 3b bysampling learned Lie algebras and applying them to transform an input image.

Original: We observe that the symmetry learned by the shallow model diverges froma clean rotation and also contains scaling and shearing.
Revised: We observe that thesymmetry learned by the shallow model diverges from a clean rotation and also contains scaling andshearing.

Original: We also see, that on rotation-MNIST, thespectrum of the network polarization matrix is zeroing not as fast as in the synthetic case.
Revised: We also see, that on rotation-MNIST, the spectrum of the network polarization matrix ismuch smoother than when trained on the synthetic data.

Original: It indicatesthat the models, although do learn the rotation symmetry, do not become fully invariant to it.
Revised: It indicates that the models, although dolearn the rotation symmetry, do not become fully invariant to it.

Original: This experiment demonstrates that our method can robustly retrieve noisy symmetries from the dataeven when an imprecise discriminator function is used.
Revised: This experiment demonstrates that our method can robustly retrieve noisy symmetries from the dataeven when an imprecise discriminator function is used.

Original: 5.4 Symmetries in networks with different configurations
Revised: 5.4 Symmetries in networks with different configurations

Original: Observing the differences in symmetrical properties for different network configurations in Experiment 5.3, we further conduct the study on how symmetry variance and symmetry bias depend onwidth, depth, and a number of parameters in the network.
Revised: Observing the differences in symmetrical properties for different network configurations in Experiment 5.3, we further conduct the study on how symmetry variance and symmetry bias depend onwidth, depth, and a number of parameters in the network.

Original: In this experiment, we are interested toknow if for some architecture it is easier to learn symmetries from the data.
Revised: In this experiment, we are interested toknow if for some architecture it is easier to learn symmetries from the data.

Original: To obtain different configurations, we sample a number of parameters from a range of [40000 , 200000]with a step of 20000 , and vary the depth of the network from 1 to 5 hidden layers.
Revised: To obtain different configurations, we sample a number of parameters from a range of [40000 , 200000]with a step of 20000 , and vary the depth of the network from 1 to 5 hidden layers.

Original: We train thenetworks until convergence on the validation split and apply our method to study how well theylearn the symmetries.
Revised: We train thenetworks until convergence on the validation split and apply our method to study how well theylearn the symmetries.

Original: We also record the accuracy of the models to correlate it with their symmetryvariance and symmetry bias.
Revised: We also record the accuracy of the models to correlate it with their symmetryvariance and symmetry bias.

Original: We report results in Figure 4.
Revised: We report results in Figure 4.

Original: From Figure 4a we can see that we make the following conclusions on how the configurations of thenetwork influence its symmetrical properties:
Revised: From Figure 4a we can see that we make the following conclusions on how the configurations of thenetwork influence its symmetrical properties:

Original: Number of parameters: we observe that the networks with more parameters learn to become moreinvariant than their shallow counterparts as indicated by consistently smaller symmetry variance.
Revised: Number of parameters: we observe that the networks with more parameters learn to become moreinvariant than their shallow counterparts as indicated by consistently smaller symmetry variance.

Original: Also,we see that the networks with fewer number of parameters generally learn less precise symmetries asshown by higher symmetry bias.
Revised: Also,we see that the networks with fewer number of parameters generally learn less precise symmetries asshown by higher symmetry bias.

Original: Depth / width: our results indicate that the sufficiently parameterized deep networks are better ableto learn invariances than the wide ones with the same amount of parameters, i.e. deeper the networkis, more invariant it becomes in the direction of the learned symmetries.
Revised: Depth / width: our results indicate that the sufficiently parameterized deep networks are better ableto learn invariances than the wide ones with the same amount of parameters, i.e. deeper the networkis, more invariant it becomes in the direction of the learned symmetries.

Original: At the same time, this doesnot hold for the quality of learned symmetry groups as indicated by fluctuating symmetry bias, whendepth > 3 .
Revised: At the same time, this doesnot hold for the quality of learned symmetry groups as indicated by fluctuating symmetry bias, whendepth > 3 .

Original: We see that the network with balanced width and depth achieve smaller symmetry biasthan the deeper ones.
Revised: We see that the network with balanced width and depth achieve smaller symmetry biasthan the deeper ones.

Original: Symmetry vs. accuracy: we also test how the symmetry variance and bias correlates with theperformance of the model.
Revised: Symmetry vs. accuracy: we also test how the symmetry variance and bias correlates with theperformance of the model.

Original: As we can see from Figure 4b, the symmetry variance and bias havemoderate correlation coefficients
Revised: As we can see from Figure 4b, the symmetry variance and bias havemoderate correlation coefficients

Original: ρ var = − 0 .
Revised: ρ var = − 0 .

Original: 49( p-value = 10 − 35 ) and ρ bias = − 0 .
Revised: 49 and ρ bias = − 0 .

Original: 31( p-value =10 − 14 ) respectively.
Revised: 31 respectively.

Original: This trend indicates that more accurate models generally also tend to have lowersymmetry variance and bias.
Revised: This trend indicatesthat more accurate models generally also tend to have lower symmetry variance and bias.

Original: However such correlation is not strict, and it is possible to have anetwork, which does not learn the most precise symmetries from the training data, but still performwell on the test.
Revised: Howeversuch correlation is not strict, and it is possible to have a network, which does not learn the mostprecise symmetries from the training data, but still perform well on the test.

Original: 5.5 Tuning the layer-wise symmetries in deep networks
Revised: 5.5 Tuning the layer-wise symmetries in deep networks

Original: In the previous experiment we analyzed how the symmetrical properties of the networks dependon the configuration of the model.
Revised: In the previous experiment we analyzed how the symmetrical properties of the networks dependon the configuration of the model.

Original: Now, we ask if by modifying training regime, we can facilitatelearning the symmetries for the network.
Revised: Now, we ask if by modifying training regime, we can facilitatelearning the symmetries for the network.

Original: In this experiment we test two known approaches to improvethe training: (i) pre-training with fine-tuning, and (ii) layer-wise training approach [40].
Revised: In this experiment we test two known approaches to improvethe training: (i) pre-training with fine-tuning, and (ii) layer-wise training approach [34].

Original: To simulate the pre-training a part of the network weights, we take randomly initialized 7 layersperceptron (mother network), and extract sub-network that consist of the first 3 layers.
Revised: To simulate the pre-training a part of the network weights, we take randomly initialized 7 layersperceptron (mother network), and extract sub-network that consist of the first 3 layers.

Original: We then attachthe classification head to this sub-network and train it independently from the mother network onrotation-MNIST dataset.
Revised: We then attachthe classification head to this sub-network and train it independently from the mother network onrotation-MNIST dataset.

Original: After that, we re-inject the learned weights into the mother network andretrain the model.
Revised: After that, we re-inject the learned weights into the mother network andretrain the model.

Original: For fine-tuning, we train all of the weights jointly.
Revised: For fine-tuning, we train all of the weights jointly.

Original: For layer-wise training, wefreeze the pre-trained part and only train the remaining layers.
Revised: For layer-wise training, wefreeze the pre-trained part and only train the remaining layers.

Original: We then apply our method to evaluatethe symmetrical properties of the models.
Revised: We then apply our method to evaluatethe symmetrical properties of the models.

Original: We plot the symmetry variance and symmetry bias aftereach layer of the network in Figure 5.
Revised: We plot the symmetry variance and symmetry bias aftereach layer of the network in Figure 5.

Original: Final and pen-ultimate layer symmetries: from Figure 5 we observe that for different trainingregimes, the network learns approximately the same symmetries at the same degree of invariance in afinal layer.
Revised: Final and pen-ultimate layer symmetries: from Figure 5 we observe that for different trainingregimes, the network learns approximately the same symmetries at the same degree of invariance in afinal layer.

Original: Surprisingly, we also noticed that in a pen-ultimate layer, the symmetries for all of thetraining regimes come very close as measured by symmetry variance and bias.
Revised: Surprisingly, we also noticed that in a pen-ultimate layer, the symmetries for all of thetraining regimes come very close as measured by symmetry variance and bias.

Original: Since the last layersare directly responsible for providing outputs for the task 1 , we hypothesize that there exist preferred optimal symmetry level for the whole network that depends on the architecture and data, but not fromthe training regime of the model .
Revised: Since the last layers are directly responsible for providing outputs for the task 1 , we hypothesize that there exist preferredoptimal symmetry level for the whole network that depends on the architecture and data, but not fromthe training regime of the model .

Original: Intermediate symmetries: on the other hand, we observed that the symmetries in the intermediateslayers of the network do depend on the training regime .
Revised: Intermediate symmetries: on the other hand, we observed that the symmetries in the intermediateslayers of the network do depend on the training regime .

Original: Firstly, the pre-trained sub-network learnsmore accurate symmetry with a higher level of invariance compared to when it is trained as a part ofthe deeper model.
Revised: Firstly, the pre-trained sub-network learnsmore accurate symmetry with a higher level of invariance compared to when it is trained as a part ofthe deeper model.

Original: We attribute it to the fact that symmetries are defined on the level of the labels,hence the invariances learned by the model in the final layer should be kept at a minimum.
Revised: We attribute it to the fact that symmetries are defined on the level of the labels,hence the invariances learned by the model in the final layer should be kept at a minimum.

Original: Secondly,after the fine-tuning or layer-wise training, symmetries in the intermediate layers of the network areboth more precise and utilized at a higher degree of invariance as indicated by smaller symmetry biasand variance respectively.
Revised: Secondly,after the fine-tuning or layer-wise training, symmetries in the intermediate layers of the network areboth more precise and utilized at a higher degree of invariance as indicated by smaller symmetry biasand variance respectively.

Original: It indicates that we can facilitate learning intermediate invariances in thenetwork by using pre-training followed by fine-tuning or layer-wise learning approach.
Revised: It indicates that we can facilitate learning intermediate invariances in thenetwork by using pre-training followed by fine-tuning or layer-wise learning approach.

Original: We assumethis insight can be useful for transfer learning when it is important to transfer as much inductive biasas possible into the other models or to other tasks.
Revised: We assumethis insight can be useful for transfer learning when it is important to transfer as much inductive biasas possible into the other models or to other tasks.

Original:  Conclusion
Revised:  Conclusion

Original: Summary We present the LieGG - an approach to retrieve the symmetries learned by a neuralnetwork.
Revised: Summary We present the LieGG - an approach to retrieve the symmetries learned by a neuralnetwork.

Original: Conveniently, our method extracts the symmetries directly in a form of corresponding
Revised: Conveniently, our method extracts the symmetries directly in a form of corresponding

Original: Lie group generators.
Revised: Lie group generators.

Original: This allows to explicitly obtain the symmetries learned by the model withoutspecifying any set of transformations of interest beforehand.
Revised: This allows to explicitly obtain the symmetries learned by the model withoutspecifying any set of transformations of interest beforehand.

Original: The LieGG also evaluates how themodel is invariant in the direction of a learned symmetry by computing the symmetry variance.
Revised: The LieGG also evaluates how themodel is invariant in the direction of a learned symmetry by computing the symmetry variance.

Original: Wemake use of our approach by studying how the quality of the symmetry learned by the networkdepends on its configuration and a training regime.
Revised: Wemake use of our approach by studying how the quality of the symmetry learned by the networkdepends on its configuration and a training regime.

Original: With this, by improving an interpretability of thesymmetry learning, we can also identify good practices that improve the symmetries in a model.
Revised: With this, by improving an interpretability of thesymmetry learning, we can also identify good practices that improve the symmetries in a model.

Original: Limitations and Future Work
Revised: Limitations and Future Work

Original: The proposed method can facilitate the interpretability of neuralnetworks and help to design efficient ways of incorporating the symmetries into the model.
Revised: The proposed method can facilitate the interpretability of neuralnetworks and help to design efficient ways of incorporating the symmetries into the model.

Original: However,some limitations remain.
Revised: However,some limitations remain.

Original: In particular, the LieGG is currently limited to the instances of GL connectedgroup, which hinders the analysis of complex symmetries, e.g. where the transformation can not bedescribed as a group action.
Revised: In particular, the LieGG is currently limited to the instances of GLconnected groups, which hinders the analysis of complex compositional symmetries.

Original: It is an intriguing direction for future research to extend the mechanismof retrieving the symmetries to more complex transformations.
Revised: It is an intriguingdirection for future research to extend the mechanism of retrieving the symmetries to more complextransformations.

Original: Nonetheless, the LieGG provides animportant building block on the way to increasing the interpretability of the symmetries learned bythe neural networks.
Revised: Nonetheless, the LieGG provides an important building block on the way toincreasing the interpretability of the symmetries learned by the neural networks.

Original: Also, in our analysis, we focus on a specific family of models, i.e. feed-forwardmulti-layer perceptions trained on the well-controlled datasets.
Revised: Also, in our analysis,we focus on a specific family of models, i.e. feed-forward multi-layer perceptions.

Original: We assume that the further large-scaleanalysis ofthe LieGG in complex scenarios and with more advanced models (e.g. graph networks,and self-attention transformers) can provide a more general understanding of the symmetry learningin deep networks.
Revised: Our work is thusthe first step toward better understanding the symmetrical properties of a wider family of models suchas convolutional networks, graph networks, and self-attention transformers.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A](d)

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [No](b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes](c)

Original: (c) Did you report error bars (e
Revised: Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes](d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]