Original: M ODEL C OMPRESSION VIA S YMMETRIES OF THE P ARAMETER S PACE
Revised: M ODEL C OMPRESSION VIA S YMMETRIES OF THE P ARAMETER S PACE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We provide a theoretical framework for neural networks in terms of the representation theory of quivers, thus revealing symmetries of the parameter space of neural networks.
Revised: We provide a theoretical framework for neural networks in terms of the representation theory of quivers, thus revealing symmetries of the parameter space of neural networks.

Original: An exploitation of these symmetries leads to a model compression algorithm for radial neural networks based on an analogue of the QR decomposition.
Revised: An exploitation of these symmetries leads to a model compression algorithm for radial neural networks based on an analogue of the QR decomposition.

Original: The algorithm is lossless; the compressed model has the same feedforward function as the original model.
Revised: The algorithm is lossless the compressed model has the same feedforward function as the original model.

Original: If applied before training, optimization of the compressed model by gradient descent is equivalent to a projected version of gradient descent on the original model
Revised: If applied before training, optimization of the compressed model by gradient descent is equivalent to a projected version of gradient descent on the original model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent work has shown that representation theory, the formal study of symmetry, provides the foundation for various innovative techniques in deep learning (Cohen & Welling, 2016) (Kondor &
Revised: Recent work has shown that representation theory, the formal study of symmetry, provides the foundation for various innovative techniques in deep learning (Cohen & Welling, 2016; Kondor &

Original: Trivedi, 2018; Ravanbakhsh et al., 2017; Cohen & Welling, 2017).
Revised: Trivedi, 2018; Ravanbakhsh et al., 2017; Cohen & Welling, 2017).

Original: Much of this previous work considers symmetries inherent to the input and output spaces, as well as distributions and functions that respect these symmetries.
Revised: Much of this previous work considers symmetries inherent to the input and output spaces, as well as distributions and functions that respect these symmetries.

Original: By contrast, in this paper, we expose a broad class of symmetries intrinsic to the parameter space of the neural networks themselves.
Revised: By contrast, in this paper, we expose a broad class of symmetries intrinsic to the parameter space of the neural networks themselves.

Original: We use these parameter space symmetries to devise a model compression algorithm that reduces the widths of the hidden layers, and hence the number of parameters.
Revised: We use these parameter space symmetries to devise a model compression algorithm that reduces the widths of the hidden layers, and hence the number of parameters.

Original: Unlike representation-theoretic techniques in the setting of equivariant neural networks, our methods are applicable to deep learning models with non-symmetric domains and non-equivariant functions, and hence pertain to some degree to all neural networks.
Revised: Unlike representation-theoretic techniques in the setting of equivariant neural networks, our methods are applicable to deep learning models with non-symmetric domains and non-equivariant functions, and hence pertain to some degree to all neural networks.

Original: Speciﬁcally, we formulate a theoretical framework for neural networks in terms of the theory of representations of quivers, a mathematical ﬁeld with connections to symplectic geometry and Lie theory (Kirillov Jr, 2016; Nakajima et al., 1998).
Revised: Speciﬁcally, we formulate a theoretical framework for neural networks inspired by quiver representation theory, a mathematical ﬁeld with connections to symplectic geometry and Lie theory (Kirillov Jr, 2016; Nakajima et al., 1998).

Original: This approach builds on that of Armenta & Jodoin (2020) and of Wood & Shawe-Taylor (1996), but is arguably simpler and encapsulates larger symmetry groups.
Revised: This approach builds on that of Armenta & Jodoin (2020) and of Wood & Shawe-Taylor (1996), but is arguably simpler and encapsulates larger symmetry groups.

Revised: Formally, a quiver is another name for a ﬁnite directed graph, and a representation of a quiver is the assignment of a vector space to each vertex and a compatible linear map to each edge.

Original: Our starting point is to regard the vector space of parameters for a neural network as a representation of a speciﬁc quiver, namely, the neural quiver (Figure 1).
Revised: Our starting point is to regard the vector space of parameters for a neural network as a representation of a particular quiver, namely, the neural quiver (Figure 1).

Original: The advantage of this viewpoint is that representations of quivers carry rich symmetry groups via change-of-basis transformations; such operations can be viewed as symmetries of the neural network parameter space. 
Revised: The advantage of this viewpoint is that representations of quivers carry rich symmetry groups via change-of-basis transformations; such operations can be viewed as symmetries of the neural network parameter space. Moreover

Original: Moreover, these symmetries may be factored out without affecting the feedforward function, leading to a lossless model compression algorithm.
Revised: [Figure] these symmetries may be factored out without affecting the feedforward function [Figure] making our method a lossless model compression algorithm

Revised: (Serra et al., 2020).

Original: Model compression has  become critically important as models have grown to billions of parameters; with compression, enormous models may be reduced and run on smaller systems with faster inference (Buciluˇa et al., 2006; Cheng et al., 2017; Frankle & Carbin, 2018; Zhang et al., 2018).
Revised: Model compression has [Figure] become critically important as models have grown to billions of parameters; with compression, enormous models may be reduced and run on smaller systems with faster inference (Buciluˇa et al., 2006; Cheng et al., 2017; Frankle & Carbin, 2018; Zhang et al., 2018).

Original: Whereas  previous approaches to model compression are based on weight pruning, quantization, matrix factorization, or knowledge distillation, we take a fundamentally different approach by exploiting symmetries of neural network parameter spaces.[Figure]
Revised: Whereas many previous approaches to model compression are based on weight pruning, quantization, matrix factorization, or knowledge distillation, similar to Sourek et al. (2020), our approach exploits symmetries of neural network parameter spaces.

Original: The size of the parameter space symmetry group is determined by properties of the activation functions.
Revised: The size of the parameter space symmetry group is determined by properties of the activation functions.

Original: We focus on radial activation functions, as in Weiler & Cesa (2019); Sabour et al.
Revised: We focus on radial activation functions, as in Weiler & Cesa (2019); Sabour et al.

Original: Weiler et al.
Revised: Weiler et al.

Original: (2018a;b); these interact favorably with certain QR decompositions, and, consequently, the model compression is signiﬁcant compared to the more common pointwise (or ‘local’) activations.
Revised: (2018a;b); these interact favorably with certain QR decompositions, and, consequently, the model compression is signiﬁcant compared to the more common pointwise (or ‘local’) activations.

Original: We refer to neural networks with radial activation functions as radial neural networks .
Revised: We refer to neural networks with radial activation functions as radial neural networks .

Original: Given a radial neural network, our results produce a new network with fewer neurons in each hidden layer and the same feedforward function, and hence the same loss for any batch of training data.
Revised: Given a radial neural network, our results produce a new network with fewer neurons in each hidden layer and the same feedforward function, and hence the same loss for any batch of training data.

Original: Moreover, the value of the loss function after a step of gradient descent applied to the compressed model is the same as the value of the loss function after a step of projected gradient descent applied to the original model.
Revised: Moreover, the value of the loss function after a step of gradient descent applied to the compressed model is the same as the value of the loss function after a step of projected gradient descent applied to the original model.

Original: As we explain below, in projected gradient descent, one subtracts a truncation of the gradient, rather than the full gradient.
Revised: As we explain below, in projected gradient descent, one subtracts a truncation of the gradient, rather than the full gradient.

Original: When the compression is signiﬁcant enough, the compressed model takes less time per epoch to train and reaches local minima faster.
Revised: When the compression is signiﬁcant enough, the compressed model takes less time per epoch to train and reaches local minima faster.

Original: To state these results slightly more precisely, recall that the parameters of a neural network with layer widths ( n 0 , n 1 , . . .
Revised: To state these results slightly more precisely, recall that the parameters of a neural network with layer widths ( n 0 , n 1 , . . .

Original: , n L ) consist of an n i × ( n i − 1 +1) matrix W i of weights for each layer
Revised: , n L ) consist of an n i × ( n i − 1 +1) matrix W i of weights for each layer

Original: i , where we include the bias as an extra column.
Revised: i , where we include the bias as an extra column.

Original: These are grouped into a tuple W = (cid:0) W i ∈ R n i × ( n i − 1 +1) (cid:1) i .
Revised: These are grouped into a tuple W = (cid:0) W i ∈ R n i × ( n i − 1 +1) (cid:1) i .

Original: We deﬁne the reduced widths recursively as n red i = min (cid:0) n i , n red i − 1 + 1 (cid:1) for i = 1 , . . .
Revised: We deﬁne the reduced widths recursively as n red i = min (cid:0) n i , n red i − 1 + 1 (cid:1) for i = 1 , . . .

Original: , L − 1 , with n red= n 0 and n red L = n L .
Revised: , L − 1 , with n red0 = n 0 and n red L = n L .

Original: Note that n red i ≤ n i for all i .
Revised: Note that n red i ≤ n i for all i .

Original: Theorem 1.1 (Informal version of Theorems 5.3 and 5.7) .
Revised: Theorem 1.1 (Informal version of Theorems 4.3 and 4.7) .

Original: Suppose a neural network has L layers with widths ( n 0 , . .
Revised: Suppose a neural network has L layers with widths ( n 0 , . .

Original: . , n L ) , parameters W , and radial activation functions.
Revised: . , n L ) , parameters W , and radial activation functions.

Original: Let f W : R n 0 → R n L be the feedforward function of the network.
Revised: Let f W : R n 0 → R n L be the feedforward function of the network.

Original: There exists  a reduced radial neural network with layer widths ( n 0 , n red1 , . . .
Revised: There exists an algorithm to produce a reduced radial neural network with layer widths ( n 0 , n red1 , . . .

Original: , n red L − 1 , n L ) , parameters R , and the same feedforward function
Revised: , n red L − 1 , n L ) , parameters R , and the same feedforward function

Original: f R = f W . 2.
Revised: f R = f W . 2.

Original: Training f R with gradient descent is an equivalent optimization problem to training f W with projected gradient descent.
Revised: Training f R with gradient descent is an equivalent optimization problem to training f W with projected gradient descent.

Original: This theorem can be interpreted as a model compression result: the reduced (or compressed) neural network R has the same accuracy as the original neural network W , and there is an explicit relationship between the gradient descent optimization problems for the two neural networks.
Revised: This theorem can be interpreted as a model compression result: the reduced (or compressed) neural network R has the same accuracy as the original neural network W , and there is an explicit relationship between the gradient descent optimization problems for the two neural networks.

Original: This result is not just theoretical; it emerges from a practical and efﬁcient algorithm which we describe  below (Algorithm 1) and implement  in Python.
Revised: This result is not just theoretical; it emerges from a practical and efﬁcient algorithm depending on successive QR decompositions. We describe this procedure below (Algorithm 1) and implement it in Python.

Original: To summarize, our contributions are as follows:
Revised: To summarize, our contributions are as follows:

Original: A theoretical framework for neural networks based on the representation theory of quivers; 2.
Revised: A theoretical framework for neural networks based on quiver representation theory; 2.

Original: A QR decomposition for radial neural networks; 3.
Revised: A QR decomposition for radial neural networks; 3.

Original: An implementation of a lossless model compression algorithm for radial neural networks; 4.
Revised: An implementation of a lossless model compression algorithm for radial neural networks; 4.

Original: A theorem relating gradient descent optimization of the original and compressed networks.
Revised: A theorem relating gradient descent optimization of the original and compressed networks.

Original: We view this work as a step in the direction of improving learning algorithms by exploiting symmetry inherent to neural network parameter spaces.
Revised: We view this work as a step in the direction of improving learning algorithms by exploiting symmetry inherent to neural network parameter spaces.

Original: As such, we expect our framework and results to generalize in several ways, including: (1) further reductions of the hidden widths, (2) incorporating certain non-radial activation functions, (3) encapsulating networks beyond MLPs, such as convolutional, recurrent, and graph neural networks, (4) integration of regularization techniques.
Revised: As such, we expect our framework and results to generalize in several ways, including: (1) further reductions of the hidden widths, (2) incorporating certain non-radial activation functions, (3) encapsulating networks beyond MLPs, such as convolutional, recurrent, and graph neural networks, (4) integration of regularization techniques.

Original: We include a detailed discussion of the limitations of our results, as well as future directions, in Section 7.
Revised: We include a detailed discussion of the limitations of our results, as well as future directions, in Section 6.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Quiver Representation Theory and Neural Networks.
Revised: Quiver Representation Theory and Neural Networks.

Original: Armenta & Jodoin (2020) give an approach to understanding neural networks in terms of quiver representations.
Revised: Armenta & Jodoin (2020) give an approach to understanding neural networks in terms of quiver representations.

Original: Our work generalizes their approach as it (1) accommodates both pointwise and non-pointwise activation functions, (2) taps into larger symmetry groups, and (3) connects more naturally to gradient descent.
Revised: Our work generalizes their approach as it (1) accommodates both pointwise and non-pointwise activation functions, (2) taps into larger symmetry groups, and (3) connects more naturally to gradient descent.

Original: Jeffreys &
Revised: Jeffreys &

Original: Lau (2021) and Manin & Marcolli (2020) also place quivers in the context of neural networks; our approach is inspired by similar algebro-geometric and categorical perspectives, but differs in our emphasis on practical consequences for optimization techniques at the core of machine learning.
Revised: Lau (2021) and Manin & Marcolli (2020) also place quivers in the context of neural networks; our approach is inspired by similar algebro-geometric and categorical perspectives, but differs in our emphasis on practical consequences for optimization techniques at the core of machine learning.

Original: Although the study of neural networks via quiver representations emerged recently, there are a number of precursors.
Revised: These works have a number of precursors.

Original: One is the studyand implementation of the “non-negative homogeneity” (also known as “positive scaling invariance”) property of ReLU activation functions (Dinh et al., 2017; Neyshabur et al., 2015; Meng et al., 2019), which is a special case of the symmetry studied in this paper.
Revised: One is the study of the “non-negative homogeneity” (also known as “positive scaling invariance”) property of ReLU activation functions (Dinh et al., 2017; Neyshabur et al., 2015; Meng et al., 2019), which is a special case of the symmetry studied in this paper.

Original: Wood & Shawe-Taylor (1996) regard layers in a neural network as representations of ﬁnite groups and consider only pointwise activation functions; by contrast, our framework captures
Revised: Wood & Shawe-Taylor (1996) regard layers in a neural network as representations of ﬁnite groups and consider only pointwise activation functions; by contrast, our framework captures

Original: Lie groups as well as non-pointwise activations.
Revised: Lie groups as well as non-pointwise activations.

Original: Our quiver approach to neural networks shares parallels with the “algebraic neural networks” of Parada-Mayorga & Ribeiro, and special cases of their formalism amount to representations of quivers over base rings beyond R , such as the ring of polynomials.
Revised: Our quiver approach to neural networks shares parallels with the “algebraic neural networks” of Parada-Mayorga & Ribeiro, and special cases of their formalism amount to representations of quivers over base rings beyond R , such as the ring of polynomials.

Original: In a somewhat different data-analytic context, Chindris & Kline (2021) use quiver representation theory in order to untangle point clouds; though they do not use neural networks.
Revised: In a somewhat different data-analytic context, Chindris & Kline (2021) use quiver representation theory in order to untangle point clouds; though they do not use neural networks.

Original: Equivariant Neural Networks.
Revised: Equivariant Neural Networks.

Original: Previously, representation theory has been used to design neural networks which incorporate symmetry as an inductive bias.
Revised: Previously, representation theory has been used to design neural networks which incorporate symmetry as an inductive bias.

Original: A variety of architectures such as
Revised: A variety of architectures such as

Original: G -convolution, steerable CNN, and Clebsch-Gordon networks are constrained by various weightsharing schemes to be equivariant or invariant to various symmetry groups (Cohen et al., 2019;
Revised: G -convolution, steerable CNN, and Clebsch-Gordon networks are constrained by various weightsharing schemes to be equivariant or invariant to various symmetry groups (Cohen et al., 2019;

Original: Weiler & Cesa, 2019; Cohen & Welling, 2016; Chidester et al., 2018; Kondor & Trivedi, 2018; Bao & Song, 2019; Worrall et al., 2017; Cohen & Welling, 2017; Weiler et al., 2018b; Dieleman et al., 2016; Lang & Weiler, 2021; Ravanbakhsh et al., 2017).
Revised: Weiler & Cesa, 2019; Cohen & Welling, 2016; Chidester et al., 2018; Kondor & Trivedi, 2018; Bao & Song, 2019; Worrall et al., 2017; Cohen & Welling, 2017; Weiler et al., 2018b; Dieleman et al., 2016; Lang & Weiler, 2021; Ravanbakhsh et al., 2017).

Original: Our approach, in contrast, does not rely on symmetry of the input domain, output space, or mapping.
Revised: Our approach, in contrast, does not rely on symmetry of the input domain, output space, or mapping.

Original: Rather, our method exploits symmetry of the parameter space and thus applies more generally to domains with no obvious symmetry.
Revised: Rather, our method exploits symmetry of the parameter space and thus applies more generally to domains with no obvious symmetry.

Original: From the point of view of model compression, equivariant networks do achieve reduction in the number of trainable parameters through weight-sharing for ﬁxed hidden dimension widths; however, in practice, they may use larger layer widths and consequently have larger memory requirements than non-equivariant models.
Revised: From the point of view of model compression, equivariant networks do achieve reduction in the number of trainable parameters through weight-sharing for ﬁxed hidden dimension widths; however, in practice, they may use larger layer widths and consequently have larger memory requirements than non-equivariant models.

Original: Sampling or summing over large symmetry groups may make equivariant models computationally slow as well (Finzi et al., 2020; Kondor & Trivedi, 2018).
Revised: Sampling or summing over large symmetry groups may make equivariant models computationally slow as well (Finzi et al., 2020; Kondor & Trivedi, 2018).

Original: Model Compression and Weight Pruning.
Revised: Model Compression and Weight Pruning.

Original: A major goal in machine learning is to ﬁnd methods for compressing models in order to reduce the number of trainable parameters, decrease memory usage, or accelerate inference and training (Cheng et al., 2017).
Revised: A major goal in machine learning is to ﬁnd methods for compressing models in order to reduce the number of trainable parameters, decrease memory usage, or accelerate inference and training (Cheng et al., 2017).

Original: Our approach toward this goal differs signiﬁcantly from most existing methods in that it is based on the inherent symmetry of neural network parameter spaces.
Revised: Our approach toward this goal differs signiﬁcantly from most existing methods in that it is based on the inherent symmetry of neural network parameter spaces.

Original: One prior model compression method is weight pruning , which removes redundant, small, or unnecessary weights from a network with little loss in accuracy (Han et al., 2015; Blalock et al., 2020; Karnin, 1990).
Revised: One prior model compression method is weight pruning , which removes redundant, small, or unnecessary weights from a network with little loss in accuracy (Han et al., 2015; Blalock et al., 2020; Karnin, 1990).

Original: Recent work has shown pruning can be done before training by identifying certain key network structures (Frankle & Carbin, 2018). 
Revised: Recent work has shown pruning can be done during training by identifying and removing weights of less relevance and reverting other weights to earlier values (Frankle & Carbin, 2018). Related work shows effective pruning may be done at initialization (Lee et al., 2019; Wang et al., 2020). Gradient-based pruning identiﬁes low saliency weights by estimating the increase in loss resulting from their removal (LeCun et al., 1990; Hassibi & Stork, 1993; Dong et al., 2017; Molchanov et al., 2016).

Original: A complementary approach is quantization , in which the bit depth of weights is decreased (Wu et al., 2016;
Revised: A complementary approach is quantization , in which the bit depth of weights is decreased (Wu et al., 2016;

Original: Howard et al., 2017;
Revised: Howard et al., 2017;

Original: Gong et al., 2014).
Revised: Gong et al., 2014).

Original: Knowledge distillation works by training a small model to mimic the performance of a larger model or ensemble of models (Buciluˇa et al., 2006; Hinton et al., 2015; Ba & Caruana, 2013).
Revised: Knowledge distillation works by training a small model to mimic the performance of a larger model or ensemble of models (Buciluˇa et al., 2006; Hinton et al., 2015; Ba & Caruana, 2013).

Original: Matrix Factorization methods replace fully connected layers with lower rank or sparse factored tensors (Cheng et al.
Revised: Matrix Factorization methods replace fully connected layers with lower rank or sparse factored tensors (Cheng et al.

Original: , 2015a;b; Tai et al., 2015; Lebedev et al., 2014;
Revised: , 2015a;b; Tai et al., 2015; Lebedev et al., 2014;

Original: Rigamonti et al., 2013; Lu et al., 2017) and can often be applied before training.
Revised: Rigamonti et al., 2013; Lu et al., 2017) and can often be applied before training.

Original: Our method involves a generalized QR decomposition, which is a type of matrix factorization; however, rather than aim for a rank reduction of linear layers, we leverage this decomposition in order to reduce hidden layer widths via change-of-basis operations on the hidden representations.
Revised: Our method involves a generalized QR decomposition, which is a type of matrix factorization; however, rather than aim for a rank reduction of linear layers, we leverage this decomposition in order to reduce hidden layer widths via change-of-basis operations on the hidden representations.

Original: P RELIMINARIES AND NOTATION

Original: In this section, we review basic notations and constructions from linear algebra, group theory, and representation theory. We provide more details in Appendices B and C. Background references include Petersen et al. (2012) and Dummit & Foote (2003). For positive integers m and n , let Hom( R m , R n ) or R m × n denote the vector space of n by m matrices with entries in R , that is, the vector space of linear functions (also known as homomorphisms ) from R m to R n .
Revised: Closest to our method are lossless compression methods.

Original: The general linear group GL n ( R ) consists of the set of invertible n by n matrices, with the operation of matrix multiplication.

Original: Equivalently, GL n ( R ) consists of all linear automorphisms of R n , with the operation of composition.

Original: Such automorphisms are given precisely by change-of-basis transformations.

Original: The unit is the identity n by n matrix, denoted id n .

Original: Let G be a group. An action (or representation ) of G on the vector space R n consists of an invertible n by n matrix ρ ( g ) for every group element g ∈ G , such that (1) for all g, h ∈ G , the matrix ρ ( gh ) is the product of the matrices ρ ( g ) and ρ ( h ) , and (2) for the identity element 1 G ∈ G , we have ρ (1 G ) = id n . In other words, an action amounts to a map of groups ρ : G → GL n ( R ) .
Revised: Serra et al.

Original: We often abbreviate ρ ( g )( v ) as simply g · v , for g ∈ G and v ∈ R n .

Original: An afﬁne map from R m to R n is one of the form x

Original: (cid:55)→ Ax + b for a n × m matrix A and a ndimensional vector b .

Original: Correspondingly, we have a n × ( m + 1) matrix W =

Original: [ b A ] .

Original: Thus, afﬁne maps from R m to R n are in bijection with points in the vector space

Original: Hom( R m +1 , R n ) . Let V = R p and let L : V → R be a differentiable function.
Revised: (2021; 2020) identify and remove stable neurons in ReLU networks.

Original: Semantically, these are the parameter space and loss function. The gradient of L at a point v ∈ V is ∇ v L [Equation] (cid:18)
Revised: Sourek et al.

Original: ∂ L [Equation] 1 (cid:12)(cid:12)(cid:12)(cid:12) v , . .
Revised: (2020) also exploit symmetry in parameter space to remove redundant neuron.

Original: ∂ L ∂x p (cid:12)(cid:12)(cid:12)(cid:12) v (cid:19) .

Original: A step of gradient descent with respect to L at learning rate η > 0 is the function [Equation] Hence, gradient descent updates the i -th coordinate v i of v to v i − η · ∂ L ∂x i | v , for i = 1 , . . . , p .
Revised: However, their symmetries are induced from permutation equivariance; ours follow from the symmetries of the radial activation.

Original: When η is clear from context, we abbreviate γ η by γ .

Original: For any k ≥ 0 , the k -fold composition of γ is γ k := γ ◦ γ ◦ · · · ◦ γ , with γ 0 being the identity map on V .

Original:  N EURAL NETWORKS AS QUIVER REPRESENTATIONS
Revised:  N EURAL NETWORKS AS QUIVER REPRESENTATIONS

Original: We turn our attention to the theory of quiver representations.
Revised: We turn our attention to quivers and their representations.

Original: Neural networks may be viewed as representations of a  quiver with the additional data of activation functions;moreover, backpropogation can also be formulated in terms of quiver representations.
Revised: Neural networks may be viewed as representations of a speciﬁc quiver with the additional data of activation functions; backpropogation can also be formulated in these terms.

Original: The advantage of this perspective is thatquiver representations come equipped witha change-of-basis symmetry group action.
Revised: The advantage of such perspectives is that representations come equipped with symmetry groups.

Original: We provide proofs and context for this section in Appendix D.
Revised: We provide proofs and context for this section in Appendix D.

Original: We begin with basic deﬁnitions. A quiver Q consists of a ﬁnite set I of vertices and a ﬁnite set E of directed edges 1 .
Revised: 3. Q UIVER REPRESENTATIONS

Original: Thus, Q includes the data of maps s : E → I and t : E → I indicating the source and target of each edge. A dimension vector forQ is a map d : I → Z ≥ 0 sending each vertex i to a non-negative integer  di = 
Revised: A quiver is a ﬁnite directed graph. A dimension vector for a quiver Q consists of a non-negative integer for every vertex, that is, a tuple d = ( d i ) i indexed by the vertices of Q .

Original: d ( i ) .
Revised: A representation of

Original: A representation of Q with dimension vector d consists of a linear map A e : R d s ( e ) → R d t ( e ) for each edge e ∈ E .
Revised: Q with dimension vector d consists of a matrix for each edge, where the matrix A e corresponding to an edge s • e −→ t • must be of size d t × d s .

Original: Therefore, a representation is a tuple  A 
Revised: In other words, a representation is a tuple of matrices ( A e ) e indexed by the edges of Q , with sizes constrained by the dimension vector. Note that each

Original: = ( A e ) e ∈ E indexed by the set of edges.
Revised: A e deﬁnes a linear map R d s → R d t .

Original: The set Rep ( Q , d )  of all possible representations of a quiver Q with dimension vector d is the direct sum of matrix spaces, and hence a vector space:
Revised: The space of representations Rep ( Q , d ) is deﬁned as the set of all possible representations of a quiver Q with dimension vector d . This space can be expressed as the product of matrix spaces: Rep ( Q , d )

Original: [Equation]
Revised: = (cid:81) e ∈ E R d t ( e ) × d s ( e ) . Here E is the edge set of Q , and s ( e ) and t ( e ) denote the source and target vertices of an edge e .

Original: 4. T HE NEURAL QUIVER
Revised: 3. T HE NEURAL QUIVER

Original: We now focus on a speciﬁc quiver.
Revised: For the remainder of the paper, we focus on a particular quiver.

Original: Let L be a positive integer.
Revised: Let L be a positive integer.

Original: The neural quiver Q L is the quiver with L + 2 vertices shown in Figure 1.
Revised: The neural quiver Q L is the quiver with L +2 vertices shown in Figure 1.

Original: The vertices in the top row are indexed from i = 0 to i
Revised: The vertices in the top row are indexed from i = 0 to i

Original: =
Revised: =

Original: L .
Revised: L .

Original: The ﬁrst vertex ( i = 0 ) in the top row is called the input vertex , the last vertex ( i = L ) is the output vertex , and every other vertex ( 1 ≤ i ≤ L − 1 ) in the top row is a hidden vertex .
Revised: The ﬁrst vertex ( i = 0 ) in the top row is called the input vertex , the last vertex ( i = L ) is the output vertex , and every other vertex ( 1 ≤ i ≤ L − 1 ) in the top row is a hidden vertex .

Original: The vertex at the bottom is called the bias vertex and is connected to each vertex in the top row except for i = 0 .
Revised: The vertex at the bottom is called the bias vertex and is connected to each vertex in the top row except for i = 0 .

Original: Notation : We exclusively consider dimension vectors whose value at the bias vertex is 1 .
Revised: When discussing representations of the neural quiver, we exclusively consider dimension vectors whose value at the bias vertex is 1 .

Original: Hence, a dimension vector for Q L will refer to an ( L + 1) -tuple of positive integers n = ( n 0 , . .
Revised: Hence, a dimension vector for the neural quiver will refer to an ( L + 1) -tuple of positive integers n = ( n 0 , . .

Original: . , n L ) .
Revised: . , n L ) .

Original: Lemma 4.1. A representation of Q L with dimension vector n is equivalent to the data of an afﬁne map R n i − 1 → R n i  for each i = 1 , . . .
Revised: With these conventions in hand, we observe that, by deﬁnition, a representation of the neural quiver with dimension vector n consists of an n i × n i − 1 matrix A i and a vector b i ∈ R n i , for each i = 1 , . . .

Original: , L .
Revised: , L .

Original: There is an isomorphism of vector spaces:
Revised: Correspondingly, there is an afﬁne map (cid:102) W i : R n i − 1 → R n i given by x (cid:55)→ A i x + b i , and an n i × (1 + n i − 1 ) matrix given by W i =

Original: [Equation]
Revised: [ b i A i ] .

Original: The idea of the proof is as follows.
Revised: As a result, we have:

Original: A representation of Q L with dimension vector n consists of a matrix A i ∈ Hom( R n i − 1 , R n i ) and a vector b i ∈ R n i , for each i = 1 , . . . , L .
Revised: Lemma 3.1.

Original: Hence, for each i , there is a corresponding afﬁne map(cid:102) W i : R n i − 1 → R n i given by x (cid:55)→ A i x + b i , and an n i × (1+ n i − 1 ) matrix given by W i = [ b i A i ] . 
Revised: A representation of Q L with dimension vector n is equivalent to the data of an afﬁne map R n i − 1 → R n i for each i = 1 , . . . , L . There is an isomorphism of vector spaces:

Original:  Notation : We henceforth denote a representation of Q L as a tuple W =
Revised: [Equation] Notation : We henceforth denote a representation of Q L as a tuple W =

Original: ( W i )
Revised: ( W i )

Original: Li =1 , where each W i belongs toHom( R1+ n i − 1 , R n i  ) , and denote the corresponding afﬁne maps as (cid:102) W i : R n i − 1 → R n i .
Revised: L i =1 , where each W i belongs to R n i × (1+ n i − 1 ) , and denote the corresponding afﬁne maps as (cid:102) W i : R n i − 1 → R n i .

Revised: 3. C HANGE - OF - BASIS SYMMETRY

Original: Recall  the  group GL n ( R ) , consisting of all invertible transformations (equivalently, change-of-basis transformations) of  the vector space R n .
Revised: Recall that the general linear group GL n ( R ) consists of the set of invertible n by n matrices. The unit is the identity n by n matrix, denoted id n .

Revised: There is a linear action of the general linear group

Revised: GL n ( R ) on the vector space R n given by matrix multiplication: (cid:126)x (cid:55)→ G(cid:126)x for G ∈ GL n ( R ) and (cid:126)x ∈ R n .

Revised: This action encodes change-of-basis transformations: the entries of the vector G(cid:126)x are the coefﬁcients when expressing the original vector (cid:126)x in the basis speciﬁed by the columns of G − 1 .

Original: Given a dimension vector n for the neural quiver,an element of the  group 
Revised: Given a dimension vector n for the neural quiver, the parameter symmetry group is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original:  consists of the choice of a transformation of R n i for each hidden vertex i , and results in a corresponding transformation of the matrices 
Revised: An element of this group consists of the choice of a transformation of R n i for each hidden vertex i , and results in a corresponding transformation of the matrices W i appearing in any representation

Original: W i appearing in any representation W with dimension vector n .
Revised: W of the neural quiver with dimension vector n .

Original: In other words, we have an action of GL hidden n on Rep ( Q L , n ) .

Original: To be explicit, a particular choice g
Revised: To be explicit, a particular choice G =

Original: = ( g i )
Revised: ( G i )

Original: L − 1 i =1  of transformations results in the following linear transformation ofrepresentations:
Revised: L − 1 i =  of transformations results in the following linear transformation of

Revised: representations 1 :

Original: [Equation]
Revised: [Equation]

Original: where g 0 = id n 0 and g L = id n L .
Revised: where G 0 = id n 0 and G L = id n L .

Original: For each i , the matrix (cid:20) [Equation] (cid:21) is block diagonal with one 1 ×block consisting of the single entry ‘ 1 ’, and one n i − 1 × n i − 1 block consisting of the inverse of the matrix g i − 1 ∈ GL n i − 1 ( R ) .
Revised: In other words: there is a linear action of the parameter symmetry group GL hidden n on the vector space Rep ( Q L , n ) of representations of the neural quiver.

Original: 4. N EURAL NETWORKS
Revised: 3. N EURAL NETWORKS

Original: For  the remainder of this paper,unless speciﬁed otherwise, we consider only the case of multilayer perceptrons (MLPs).
Revised: For simplicity, in the body of this paper, we focus on the most basic neural network architecture, namely that of multi-layer perceptrons (MLPs).

Original:  We expect our methods to work for more general architectures as well. For clarity, we establish precisely what we consider to be part of a neural network.
Revised: These the neural networks whose underlying architecture is speciﬁed by the neural quiver. We include a discussion of more complex quivers and architectures in the appendix. We begin with the following somewhat abstract deﬁnition:

Original: Deﬁnition 4.2.
Revised: Deﬁnition 3.2.

Original: An MLP neural network consists of the following:
Revised: An Q L - neural network consists of the following:

Original: Hyperparameters.
Revised: Hyperparameters.

Original: A positive integer L and a dimension vector n = ( n 0 , n 1 , n 2 , . .
Revised: A positive integer L and a dimension vector n = ( n 0 , n 1 , n 2 , . .

Original: . , n L ) for the neural quiver Q L . 2.
Revised: . , n L ) for the neural quiver Q L . 2.

Original: Trainable parameters.
Revised: Trainable parameters.

Original: A representation W =
Revised: A representation W =

Original: ( W 1 , . . .
Revised: ( W 1 , . . .

Original: , W L ) of the quiver Q L with dimension vector n .
Revised: , W L ) of the neural Q L with dimension vector n .

Original: So, for i = 1 , . . . , L , we have a matrix W i ∈ Hom( R1+ n i − 1 , R n i  ) .
Revised: So each W i is a matrix in R n i × (1+ n i − 1 ) .

Original: Activation functions.
Revised: Activation functions.

Original: Piece-wise differentiable functions a i : R n i → R n i for i = 1 , . . .
Revised: Piece-wise differentiable functions a i : R n i → R n i for i = 1 , . . .

Original: , L .
Revised: , L .

Original: These are grouped into a tuple a =
Revised: These are grouped into a tuple a =

Original: ( a 1 , . .
Revised: ( a 1 , . .

Original: . , a L ) .
Revised: . , a L ) .

Revised: For the remainder of this paper, unless speciﬁed otherwise, “neural network” will refer to “ Q Lneural network”.

Revised: We emphasize, however, that there is a general notion of a Q -neural network for any acyclic quiver Q with unique sink (see Appendix D).

Revised: Additionally, as indicated above, in the case of the neural quiver, a Q L -neural network is equivalent to an MLP.

Original: We denote by Neur ( Q L , n ) the vector space of neural networks with L layers and dimension vector n , and regard elements therein as pairs ( W , a ) .
Revised: We denote by Neur ( Q L , n ) the vector space of neural networks with L layers and dimension vector n , and regard elements therein as pairs ( W , a ) .

Original: The neural function (or feedforward function ) of a neural network ( W , a ) is deﬁned as:
Revised: The neural function (or feedforward function ) of a Q L -neural network ( W , a ) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:102) W i : R n i − 1 → R n i is the afﬁne map corresponding to W i (see Section 4.1).
Revised: where (cid:102) W i : R n i − 1 → R n i is the afﬁne map corresponding to W i (see Section 3.2).

Original: We now formulate the optimization problem for neural networks  in the language of quiver representations.
Revised: The optimization problem for neural networks can be formulated in terms of representations of the neural quiver.

Original: We ﬁx a dimension vector n for Q L , and activation functions a .
Revised: Fix a dimension vector n for Q L , and activation functions a .

Original: To a batch of training data { ( x j , y j ) } ⊆ R n 0 × R n L , there is an associated loss function on  Rep ( Q L , n )  deﬁned as
Revised: To a batch of training data { ( (cid:126)x j , (cid:126)y j ) } ⊆ R n 0 × R n L , the associated loss function on the space Rep ( Q L , n ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where C : R n L × R n L → R is a cost function.
Revised: where C : R n L × R n L → R is a cost function.

Original: As in Section 3, there is a gradient descent mapγ : Rep ( Q L , n ) →
Revised: The map

Original: Rep ( Q L , n ) for any learning rate.
Revised: [Equation] performs one step of gradient descent with learning rate η .

Original: Hence, the backpropogation algorithm can be regarded as taking place on thevector space Rep ( Q L , n ) of representations of the neural quiver Q L .
Revised: Hence, the backpropogation algorithm can be regarded as taking place on the space Rep ( Q L , n ) of representations of the neural quiver.

Original:  M AIN THEORETICAL RESULTS
Revised:  M AIN THEORETICAL RESULTS

Original: In this section, we ﬁrst deﬁne radial neural networks.
Revised: In this section, we ﬁrst deﬁne radial neural networks.

Original: Then, adopting the framework developed in the previous section, we proceed to state a QR decomposition for such networks, present an algorithm to compute the decomposition, and relate the decomposition to gradient descent.
Revised: Then, adopting the framework developed in the previous section, we proceed to state a QR decomposition for such networks, present an algorithm to compute the decomposition, and relate the decomposition to gradient descent.

Original: 5. R ADIAL NEURAL NETWORKS
Revised: 4. R ADIAL NEURAL NETWORKS

Original: We begin by stating the deﬁnition of radial functions.
Revised: We begin by stating the deﬁnition of radial functions.

Original: A piece-wise differentiable function a : R n → R n is called radial if a ( v ) =
Revised: A piece-wise differentiable function a : R n → R n is called radial if a ( v ) =

Original: h ( | v | ) v | v | for some h : R → R .
Revised: h ( | v | ) v | v | for some h : R → R .

Original: Hence, a sends each input vector v ∈ R n to a scalar multiple of itself, and that scalar depends only on the norm of the vector.
Revised: Hence, a sends each input vector v ∈ R n to a scalar multiple of itself, and that scalar depends only on the norm of the vector.

Original: Example 5.1.
Revised: Example 4.1.

Original: (1) The squashing function, where h ( r ) =
Revised: (1) The squashing function, where h ( r ) =

Original: r 2 r 2 +1 .
Revised: r 2 r 2 +1 .

Original: (2) Shifted ReLU, where h ( r ) =
Revised: (2) Shifted ReLU, where h ( r ) =

Original: ReLU ( r − b ) for  apositive real number b .
Revised: ReLU ( r − b ) for r > 0 a real number b .

Original: See Weiler & Cesa (2019) and the references therein for more examples and discussion of radial functions. 
Revised: See Weiler & Cesa (2019) and the references therein for more examples and discussion of radial functions. See Figure

Original: Let n be a dimension vector for the neural quiver Q L .
Revised: Let n be a dimension vector for the neural quiver Q L .

Original: A neural network ( W , a ) ∈
Revised: A neural network ( W , a ) ∈

Original: Neur ( Q L , n ) is a radial neural network if each a i is a radial function on R n i .
Revised: Neur ( Q L , n ) is a radial neural network if each a i is a radial function on R n i .

Original: Hence, there are functions
Revised: Hence, there are functions

Original: h i : R → R such that a i ( v ) =
Revised: h i : R → R such that a i ( v ) =

Original: h i ( | v | ) v | v | for each i .
Revised: h i ( | v | ) v | v | for each i .

Original: Let O ( n ) denote the following product of the orthogonal groups:
Revised: Let O ( n ) denote the following product of the orthogonal groups:

Original: [Equation]
Revised: [Equation]

Original: Note that O ( n ) is a subgroup of  GL hidden n and hence acts on Rep ( Q , n ) .
Revised: Note that O ( n ) is a subgroup of the parameter symmetry group GL hidden n and hence acts on Rep ( Q , n ) .

Original: Radial functions have the ﬂexibility of deﬁning the activation functions of a neural network with arbitrary dimension vector.
Revised: Radial functions have the ﬂexibility of deﬁning the activation functions of a neural network with arbitrary dimension vector.

Original: To be more explicit:
Revised: To be more explicit:

Original: Construction 5.2.
Revised: Construction 4.2.

Original: Suppose h
Revised: Suppose h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: , h L ) is a tuple of functions 2h i : R → R , and suppose n= ( n 0 , . . . , n L ) is a dimension vector for Q L .
Revised: , h L ) is a tuple of functions 2 R → R , and suppose n is a dimension vector for Q L .

Original: Consider the tuple of activation functions: 
Revised: Consider the tuple of activation functions: a ( n ) h

Original: [Equation]
Revised: [Equation]

Original: Any representation W of Q L with dimension vector n deﬁnes a neural network (cid:16) W , a ( n ) h (cid:17) as in
Revised: Any representation W of Q L with dimension vector n deﬁnes a neural network (cid:16) W , a ( n ) h (cid:17) as in

Original: Deﬁnition 4.2.
Revised: Deﬁnition 3.2.

Original: We write simply ( W , h ) for this network and f ( W , h ) for its neural function.
Revised: We write simply ( W , h ) for this network and f ( W , h ) for its neural function.

Original: 5. T HE QR DECOMPOSITION
Revised: 4. T HE QR DECOMPOSITION

Original: We now formulate the QR decomposition for representations of the neural quiver, which interacts favorably with radial neural networks.
Revised: We now formulate the QR decomposition for representations of the neural quiver, which interacts favorably with radial neural networks.

Original: Proofs and further context appear in Appendix E.
Revised: Proofs and further context appear in Appendix E.

Original: The ﬁrst step is to associate a ‘reduced’ dimension vector to each dimension vector of the neural quiver.
Revised: The ﬁrst step is to associate a ‘reduced’ dimension vector to each dimension vector of the neural quiver.

Original: Let n = ( n 0 , n 1 , . .
Revised: Let n = ( n 0 , n 1 , . .

Original: . , n L ) be a dimension vector for Q L .
Revised: . , n L ) be a dimension vector for Q L .

Original: Its reduction n red = ( n red0 , n red1 , . .
Revised: Its reduction n red = ( n red0 , n red1 , . .

Original: . , n red L ) is deﬁned recursively 3 by setting n red0 = n 0 , then n red i =
Revised: . , n red L ) is deﬁned recursively 3 by setting n red0 = n 0 , then n red i =

Original: min( n i , n red i − 1 + 1) for i = 1 , . . .
Revised: min( n i , n red i − 1 + 1) for i = 1 , . . .

Original: , L − 1 , and ﬁnally n red L = n L .
Revised: , L − 1 , and ﬁnally n red L = n L .

Original: We abbreviate Rep ( Q L , n ) by Rep ( n ) and
Revised: We abbreviate Rep ( Q L , n ) by Rep ( n ) and

Original: Rep ( Q L , n red ) by Rep ( n red ) .
Revised: Rep ( Q L , n red ) by Rep ( n red ) .

Original: Observe that n red i ≤ n i for each i .
Revised: Observe that n red i ≤ n i for each i .

Original: Therefore, taking a representation with dimension vector n red , i.e., a certain tuple of matrices, one can pad each matrix with the necessary number of rows of zeros below and columns of zeros on the right in order to produce a representation with dimension vector n .
Revised: Therefore, taking a representation with dimension vector n red , i.e., a certain tuple of matrices, one can pad each matrix with the necessary number of rows of zeros below and columns of zeros on the right in order to produce a representation with dimension vector n .

Original: Thus we have an inclusion: ι : Rep ( n red ) (cid:44) →
Revised: Thus we have an inclusion: ι : Rep ( n red ) (cid:44) →

Original: Rep ( n ) .
Revised: Rep ( n ) .

Original: Fix a tuple h
Revised: Fix a tuple h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: , h L ) of functions
Revised: , h L ) of functions

Original: h i : R → R .
Revised: h i : R → R .

Original: As in Construction 5.2, we attach a radial neural network (and hence a neural function) to every representation of Q L .
Revised: As in Construction 4.2, we attach a radial neural network (and hence a neural function) to every representation of Q L .

Original: Theorem 5.3.
Revised: Theorem 4.3.

Original: Let n be a dimension vector and ﬁx h as above.
Revised: Let n be a dimension vector and ﬁx h as above.

Original: For any W ∈ Rep ( n ) there exist:
Revised: For any W ∈ Rep ( n ) there exist:

Original: [Equation]
Revised: Q ∈ O ( n ) , R = ( R 1 , . . . , R L ) ∈

Original:  and
Revised: Rep ( n red ) , and

Original: [Equation] such that:
Revised: U ∈ Rep ( n ) , such that:

Original: The matrices R 1 , . . .
Revised: The matrices R 1 , . . .

Original: , R L − 1 are upper triangular.
Revised: , R L − 1 are upper triangular.

Original: The following equality holds: [Equation]
Revised: The following equality holds: W = Q · R + U 3.

Original: The networks deﬁned by W and R have identical neural functions: f ( W , h )
Revised: The networks deﬁned by W and R have identical neural functions: f ( W , h )

Original: =
Revised: =

Original: f ( R , h ) 
Revised: f ( R , h ) .

Original:  In the the second point above, we identify R with its image in Rep ( n ) under ι , and invoke the action of O ( n ) on Rep ( n ) .
Revised: exists In the the second point above, we identify R with its image in Rep ( n ) under ι , and invoke the action of O ( n ) on Rep ( n ) .

Original: We say that R is the reduced representation corresponding to W .
Revised: We say that R is the reduced representation corresponding to W .

Original: The proof of Theorem 5.3 (appearing in Appendix E.2) relies on Algorithm 1 – a constructive algorithm proceeding layer by layer and computing a QR decomposition at each stage.
Revised: The proof of Theorem 4.3 (appearing in Appendix E.2) relies on Algorithm 1 – a constructive algorithm proceeding layer by layer and computing a QR decomposition at each stage.

Original: Remark 5.4.
Revised: Remark 4.4.

Original: Part (2) of Theorem 5.3 is a quiver representation analogue of the usual QR decomposition (c.f. Section B.2); indeed, each matrix (except possibly the last) in R is upper triangular and
Revised: Part (2) of Theorem 4.3 is a quiver representation analogue of the usual QR decomposition (c.f. Section B.2); indeed, each matrix (except possibly the last) in R is upper triangular and

Original: Q is a tuple of orthogonal matrices.
Revised: Q is a tuple of orthogonal matrices.

Original: Algorithm1: input : W ∈ Rep ( n ) output : Q , R , U from Theorem 5.
Revised: Algorithm input : W ∈ Rep ( n ) output : Q , R , U from Theorem 4.

Original: Q , R , U ← [ ] , [ ] , [0] // initialize output matrix lists
Revised: Q , R , U ← [ ] , [ ] , [0] // initialize output matrix lists

Original: V 1 ← W// next layer transformed weights for i ← 1 to L − 1 do // iterate through layers if n red i < n i then
Revised: V 1 ← W// next layer transformed weights for i ← 1 to L − 1 do // iterate through layers if n red i < n i then

Original: Q i , R i ← QR-decomp( V i , mode = complete ) //
Revised: Q i , R i ← QR-decomp( V i , mode = complete ) //

Original: V i
Revised: V i

Original: =
Revised: =

Original: Q
Revised: Q

Original: i ◦ inc i ◦
Revised: i ◦ inc i ◦

Original: R i else
Revised: R i else

Original: Q i , R i ← QR-decomp( V i ) // V i
Revised: Q i , R i ← QR-decomp( V i ) // V i

Original: =
Revised: =

Original: Q
Revised: Q

Original: i ◦
Revised: i ◦

Original: R
Revised: R

Original: i end
Revised: i end

Original: Append Q i to Q
Revised: Append Q i to Q

Original: Append R i to R // reduced weights for layer
Revised: Append R i to R // reduced weights for layer

Original: i
Revised: i

Original: Append W
Revised: Append W

Original: i +1 ◦
Revised: i +1 ◦

Original: (cid:20) 

Revised: (cid:20)

Original: Q i (cid:21) ◦ p i ◦ (cid:20) 

Revised: Q i (cid:21) ◦ p i ◦ (cid:20)

Original: Q − 1 i (cid:21) to U // matrix multiplication
Revised: Q − 1 i (cid:21) to U // matrix multiplication

Original: Set V i +1 ← W
Revised: Set V i +1 ← W

Original: i +1 ◦
Revised: i +1 ◦

Original: (cid:20) 

Revised: (cid:20)

Original: Q i (cid:21) ◦ (cid:102) inc i // transform next layer end
Revised: Q i (cid:21) ◦ (cid:102) inc i // transform next layer end

Original: Append V L to R return Q , R , and U
Revised: Append V L to R return Q , R , and U

Original: Notation : The symbol ‘ ◦ ’ denotes matrix multiplication.
Revised: Notation : The symbol ‘ ◦ ’ denotes matrix multiplication.

Original: We initialize U as a list with a single element: the zero n 1 × (1 + n 0 ) matrix.
Revised: We initialize U as a list with a single element: the zero n 1 × (1 + n 0 ) matrix.

Original: For i = 1 , . . .
Revised: For i = 1 , . . .

Original: , L − 1 , we have the standard inclusions inc i = inc n red
Revised: , L − 1 , we have the standard inclusions inc i = inc n red

Original: i ,n i : R n red i (cid:44) → R n i and (cid:102) inc i = inc 1+ n red
Revised: i ,n i : R n red i (cid:44) → R n i and (cid:102) inc i = inc 1+ n red

Original: i , 1+ n i : R 1+ n red
Revised: i , 1+ n i : R 1+ n red

Original: i (cid:44) → R 1+ n i into the ﬁrst n red i and 1 + n red i coordinates, respectively.
Revised: i (cid:44) → R 1+ n i into the ﬁrst n red i and 1 + n red i coordinates, respectively.

Original: As matrices, they have ones along the main diagonal and zeros elsewhere.
Revised: As matrices, they have ones along the main diagonal and zeros elsewhere.

Original: The map p i : R n i +

Revised: The map p i : R n i +

Original: → R n i +1 zeros the ﬁrst 1 + n red i coordinates and leaves the remaining coordinates unchanged.
Revised: → R n i +1 zeros the ﬁrst 1 + n red i coordinates and leaves the remaining coordinates unchanged.

Original: As a matrix, p i is block diagonal with two square blocks; the top block is the zero matrix of size n red i + 1 , and the bottom is the identity matrix of size n i − n red i .
Revised: As a matrix, p i is block diagonal with two square blocks; the top block is the zero matrix of size n red i + 1 , and the bottom is the identity matrix of size n i − n red i .

Original: By deﬁnition, either n red i = n red i − 1 + 1 or n red i = n i .
Revised: By deﬁnition, either n red i = n red i − 1 + 1 or n red i = n i .

Original: In the former case, n red i − 1 + 1 ≤ n i and
Revised: In the former case, n red i − 1 + 1 ≤ n i and

Original: QR-decomp with ‘ mode = complete ’ computes the QR decomposition of the n i × (1 + n red i − 1 ) matrix V i as Q i ◦ inc i ◦
Revised: QR-decomp with ‘ mode = complete ’ computes the QR decomposition of the n i × (1 + n red i − 1 ) matrix V i as Q i ◦ inc i ◦

Original: R i where Q i ∈
Revised: R i where Q i ∈

Original: O ( n i ) and R i is upper-triangular of size n red
Revised: O ( n i ) and R i is upper-triangular of size n red

Original: i × n red i .
Revised: i × n red i .

Original: In the latter case, n red i − 1 +1 ≥ n i and QR-decomp computes the QR decomposition of the n i × (1+ n red i − 1 ) matrix V i as Q i ◦
Revised: In the latter case, n red i − 1 +1 ≥ n i and QR-decomp computes the QR decomposition of the n i × (1+ n red i − 1 ) matrix V i as Q i ◦

Original: R i where Q i ∈
Revised: R i where Q i ∈

Original: O ( n i ) and R i is upper-triangular of size n i × (1 + n red i − 1 ) .
Revised: O ( n i ) and R i is upper-triangular of size n i × (1 + n red i − 1 ) .

Original: R
Revised: R

Original: R 4 R R 4 R
Revised: R 4 R R 4 R

Original: R  1 . 20 .
Revised: R  1 . 20 .

Original: 5 − .
Revised: 5 − .

Original: 1  σ  0 .
Revised: 1  σ  0 .

Original: 7 − .
Revised: 7 − .

Original: 4 − .
Revised: 4 − .

Original: 2 − .
Revised: 2 − .

Original: 5 − .
Revised: 5 − .

Original: 4  σ (cid:0) .
Revised: 4  σ (cid:0) .

Original: 7 . 4 (cid:1)
Revised: 7 . 4 (cid:1)

Original: R
Revised: R

Original: R R
Revised: R R

Original: R 4 R
Revised: R 4 R

Original: R (cid:0) − 1 . 31 (cid:1) σ  − 0 .
Revised: R (cid:0) − 1 . 31 (cid:1) σ  − 0 .

Original: 2 − 0 .
Revised: 2 − 0 .

Original: 3 − 0 .
Revised: 3 − 0 .

Original: 1 − 1 .
Revised: 1 − 1 .

Original: 1  σ (cid:0) .
Revised: 1  σ (cid:0) .

Original: 7 . 4 (cid:1)
Revised: 7 . 4 (cid:1)

Original: R
Revised: R

Original: R R R R
Revised: R R R R

Original: R (cid:0) − 1 . 31 (cid:1) σ (cid:0) 1 . 17 (cid:1) σ (cid:0) − 0 .
Revised: R (cid:0) − 1 . 31 (cid:1) σ (cid:0) 1 . 17 (cid:1) σ (cid:0) − 0 .

Original: 5 (cid:1)
Revised: 5 (cid:1)

Original: Example 5.5.
Revised: Example 4.5.

Original: Suppose the dimension vector is n = (1 , 8 , 16 , 8 , 1) , so the original radial neural network model has (cid:80) 4 i =1 ( n i − 1 + 1) n i = 305 trainable parameters.
Revised: Suppose the dimension vector is n = (1 , 8 , 16 , 8 , 1) , so the original radial neural network model has (cid:80) 4 i =1 ( n i − 1 + 1) n i = 305 trainable parameters.

Original: The reduced dimension vector is n red = (1 , 2 , 3 , 4 , 1) .
Revised: The reduced dimension vector is n red = (1 , 2 , 3 , 4 , 1) .

Original: One computes that the compressed model has (cid:80) 4 i =1 ( n red i − 1 + 1)( n red i ) =
Revised: One computes that the compressed model has (cid:80) 4 i =1 ( n red i − 1 + 1)( n red i ) =

Original: trainable parameters.
Revised: trainable parameters.

Original: Another example appears in Figure 2.
Revised: Another example appears in Figure 3.

Original: 5. P ROJECTED GRADIENT DESCENT
Revised: 4. P ROJECTED GRADIENT DESCENT

Original: In this section, we use the QR decomposition of Theorem 5.3 to relate projected gradient descent for a representation to (usual) gradient descent for the corresponding reduced representation.
Revised: In this section, we use the QR decomposition of Theorem 4.3 to relate projected gradient descent for a representation to (usual) gradient descent for the corresponding reduced representation.

Original: Proofs and further context appear in Appendix F.
Revised: Proofs and further context appear in Appendix F.

Original: We adopt the notation of the previous sections.
Revised: We adopt the notation of the previous sections.

Original: In particular, we ﬁx a tuple h
Revised: In particular, we ﬁx a tuple h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: , h L ) ; as in
Revised: , h L ) ; as in

Original: Construction 5.2, this can be used to deﬁne a neural network for any representation of Q L , with any dimension vector.
Revised: Construction 4.2, this can be used to deﬁne a neural network for any representation of Q L , with any dimension vector.

Original: Furthermore, we ﬁx a batch of training data in R n 0 × R n L .
Revised: Furthermore, we ﬁx a batch of training data in R n 0 × R n L .

Original: Using h , we obtain loss functions L : Rep ( n ) → R and L red : Rep ( n red ) → R (see Section 4.2).
Revised: Using h , we obtain loss functions L : Rep ( n ) → R and L red : Rep ( n red ) → R (see Section 3.4).

Original: For a ﬁxed learning rate, there are resulting  gradient descent maps given by:
Revised: For a ﬁxed learning rate, there are resulting one-step gradient descent maps given by:

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 5.6.
Revised: Deﬁnition 4.6.

Original: Deﬁne Pr : Rep ( n ) → Rep ( n ) to be the map that, given a representation T ∈
Revised: Deﬁne Pr : Rep ( n ) → Rep ( n ) to be the map that, given a representation T ∈

Original: Rep ( n ) , zeros the bottom left ( n i − n red i ) × (1 + n red i − 1 ) submatrix of T i , for each i .
Revised: Rep ( n ) , zeros the bottom left ( n i − n red i ) × (1 + n red i − 1 ) submatrix of T i , for each i .

Original: Schematically,
Revised: Schematically,

Original: Pr :
Revised: Pr :

Original: T = (cid:18)(cid:20) ∗ ∗ ∗ ∗ (cid:21)(cid:19)
Revised: T = (cid:18)(cid:20) ∗ ∗ ∗ ∗ (cid:21)(cid:19)

Original: L i = (cid:55)−→
Revised: L i = (cid:55)−→

Original: Pr ( T ) =
Revised: Pr ( T ) =

Original: (cid:18)(cid:20) ∗ ∗ 0 ∗ (cid:21)(cid:19)
Revised: (cid:18)(cid:20) ∗ ∗ 0 ∗ (cid:21)(cid:19)

Original: L i = .
Revised: L i = .

Original: The  projected gradient descent map on Rep ( n ) at learning rate η > 0 is deﬁned as:
Revised: The one-step projected gradient descent map on Rep ( n ) at learning rate η > 0 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Hence, while all entries of each matrix W i in the representation W contribute to the computation of the gradient ∇ W L , only those not in the bottom left submatrix get updated under the projected gradient descent map γ proj .
Revised: Hence, while all entries of each matrix W i in the representation W contribute to the computation of the gradient ∇ W L , only those not in the bottom left submatrix get updated under the projected gradient descent map γ proj .

Original: To ease notation in the statement of the following result, we identify elements of Rep ( n red ) with their images in Rep ( n ) under ι .
Revised: To ease notation in the statement of the following result, we identify elements of Rep ( n red ) with their images in Rep ( n ) under ι .

Original: Theorem 5.7.
Revised: Theorem 4.7.

Original: Let W ∈ Rep ( n ) and let Q , R , and U be the outputs of Algorithm 1.
Revised: Let W ∈ Rep ( n ) and let Q , R , and U be the outputs of Algorithm 1.

Original: Fix a learning rate η > 0 .
Revised: Fix a learning rate η > 0 .

Original: For any k ≥ 0 , we have:
Revised: For any k ≥ 0 , we have:

Original: [Equation]
Revised: [Equation]

Original:  While the proof of the second statement of Theorem 5.7 is quite technical, the proof of the ﬁrst relies on a basic interaction of orthogonal transformations with gradient descent. We brieﬂy illustratethis interaction (Appendices C and F contain more details).
Revised: and [Equation] While the proof of the second statement of Theorem 4.7 is quite technical, the proof of the ﬁrst relies on a basic interaction of orthogonal transformations with gradient descent, which we brieﬂy illustrate (Appendices C and F contain more details).

Original: As in Section 3, let L : R p → R , and let γ : R p
Revised: Let L : R p → R be a function, and let γ : R p

Original: → R p be the gradient descent map with respect to L .
Revised: → R p be the gradient descent map with respect to L .

Original: Suppose Q ∈ O ( p ) is an orthogonal transformation of R p that preserves the level sets of L , that is, L ( Q ( v ))
Revised: Suppose Q ∈ O ( p ) is an orthogonal transformation of R p that preserves the level sets of L , that is, L ( Q ( v ))

Original: =
Revised: =

Original: L ( v ) for all v ∈ R p .
Revised: L ( v ) for all v ∈ R p .

Original: Then Q commutes with gradient descent, i.e. γ ( Q ( v ))
Revised: Then Q commutes with gradient descent, i.e. γ ( Q ( v ))

Original: =
Revised: =

Original: Q ( γ ( v )) for all v ∈ R p .
Revised: Q ( γ ( v )) for all v ∈ R p .

Original: See Figure 3.
Revised: See Figure 4.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In addition to the theoretical results in this work, we provide an implementation of QRDimRed as described in Algorithm 1.
Revised: In addition to the theoretical results in this work, we provide an implementation of QRDimRed as described in Algorithm 1.

Original: We (1) empirically validate that our implementation satisﬁes the claims of Theorems 5.3 and Theorem 5.7 and (2) quantify real-world performance.
Revised: We (1) empirically validate that our implementation satisﬁes the claims of Theorems 4.3 and Theorem 4.7 and (2) quantify real-world performance.

Original: Empirical veriﬁcation of Theorem 5.3 We verify the Theorem using a small model and synthetic data.
Revised: Empirical veriﬁcation of Theorem 4.3 We verify the Theorem using a small model and synthetic data.

Original: We learn the function f ( x ) =
Revised: We learn the function f ( x ) =

Original: e − x 2 using N = 121 samples x j = − 3+ j/ 20 for 0 ≤ j < 121 .
Revised: e − x 2 using N = 121 samples x j = − 3+ j/ 20 for 0 ≤ j < 121 .

Original: We model f W as a radial neural network with layer widths n = (1 , 6 , 7 , 1) and activation the radial shifted sigmoid h ( x ) =
Revised: We model f W as a radial neural network with layer widths n = (1 , 6 , 7 , 1) and activation the radial shifted sigmoid h ( x ) =

Original: 1 / (1+ e − x + b ) .
Revised: 1 / (1+ e − x + b ) .

Original: Applying QRDimRed gives a radial neural network f R with widths n red =
Revised: Applying QRDimRed gives a radial neural network f R with widths n red =

Original: Theorem 5.3 implies that the neural functions of f W and f R are equal.
Revised: Theorem 4.3 implies that the neural functions of f W and f R are equal.

Original: Over 10 random initializations of W , the mean absolute error (1 /N )
Revised: Over 10 random initializations of W , the mean absolute error (1 /N )

Original: (cid:80) j | f W ( x j ) − f R ( x j ) | = 1 . 31 · 10 − 8 ± 4 .
Revised: (cid:80) j | f W ( x j ) − f R ( x j ) | = 1 . 31 · 10 − 8 ± 4 .

Original: 45 · 10 − 9 .
Revised: 45 · 10 − 9 .

Original: Thus f W and f R agree up to machine precision.
Revised: Thus f W and f R agree up to machine precision.

Original: Empirical veriﬁcation of Theorem 5.7 Similarly, we verify the conclusions of Theorem 5.7 using synthetic data.
Revised: Empirical veriﬁcation of Theorem 4.7 Similarly, we verify the conclusions of Theorem 4.7 using synthetic data.

Original: The claim is that training f Q − 1 · W with objective L by projected gradient descent coincides with training f R with objective L red by usual gradient descent.
Revised: The claim is that training f Q − 1 · W with objective L by projected gradient descent coincides with training f R with objective L red by usual gradient descent.

Original: We veriﬁed this forepochs at learning rate 0.01.
Revised: We veriﬁed this forepochs at learning rate 0.01.

Original: Over 10 random initializations of W , the loss functions match up to machine precision with |L − L red | = 4 .
Revised: Over 10 random initializations of W , the loss functions match up to machine precision with |L − L red | = 4 .

Original: 02 · 10 − 9 ± 7 . 01 · 10 − 9 .
Revised: 02 · 10 − 9 ± 7 . 01 · 10 − 9 .

Original: Reduced model trains faster.
Revised: Reduced model trains faster.

Original: Due to the relation between projected gradient descent of the full network W and gradient descent of the reduced network R , our method may be applied before training to produce a smaller model class which trains faster without sacriﬁcing accuracy.
Revised: Due to the relation between projected gradient descent of the full network W and gradient descent of the reduced network R , our method may be applied before training to produce a smaller model class which trains faster without sacriﬁcing accuracy.

Original: We test this hypothesis in learning the function f : R 2 → R 2 sending x = ( t 1 , t 2 ) to ( e − t 21 , e − t 22 ) using N = 121 2 samples ( − 3+ j/ 20 , − 3+ k/ 20) for 0 ≤ j, k < 121 .
Revised: We test this hypothesis in learning the function f : R 2 → R 2 sending x = ( t 1 , t 2 ) to ( e − t 21 , e − t 22 ) using N = 121 2 samples ( − 3+ j/ 20 , − 3+ k/ 20) for 0 ≤ j, k < 121 .

Original: We model f W as a radial neural network with layer widths n = (2 , 16 , 64 , 128 , 16 , 2) and activation the radial sigmoid h ( r ) =
Revised: We model f W as a radial neural network with layer widths n = (2 , 16 , 64 , 128 , 16 , 2) and activation the radial sigmoid h ( r ) =

Original: 1 / (1 + e − r ) .
Revised: 1 / (1 + e − r ) .

Original: Applying QRDimRed gives a radial neural network f R with widths n red =
Revised: Applying QRDimRed gives a radial neural network f R with widths n red =

Original: We trained both models until the training loss was ≤ 0 . 01 .
Revised: We trained both models until the training loss was ≤ 0 . 01 .

Original: Running on a system with an Intel i5-8257U@1.40GHz and 8GB of RAM and averaged over 10 random initializations, the reduced network trained in 15 .
Revised: Running on a system with an Intel i5-8257U@1.40GHz and 8GB of RAM and averaged over 10 random initializations, the reduced network trained in 15 .

Original: 32 ± 2 .
Revised: 32 ± 2 .

Original: 53 seconds and the original network trained in 31 .
Revised: 53 seconds and the original network trained in 31 .

Original: 24 ± 4 .
Revised: 24 ± 4 .

Original: 55 seconds.
Revised: 55 seconds.

Original:  C ONCLUSIONS AND FUTURE WORK
Revised:  C ONCLUSIONS AND FUTURE WORK

Original: In this paper, we have adopted the formalism of quiver representation theory in order to establish a theoretical framework for neural networks.
Revised: In this paper, we have adopted the formalism of quiver representation theory in order to establish a theoretical framework for neural networks.

Original: While drawing inspiration from previous work, our approach is novel in that it (1) reveals a large group of symmetries of neural network parameter spaces, (2) leads to a version of the QR decomposition in the context of radial neural networks, and (3) precipitates an algorithm to reduce the widths of the hidden layers in such networks.
Revised: While drawing inspiration from previous work, our approach is novel in that it (1) reveals a large group of symmetries of neural network parameter spaces, (2) leads to a version of the QR decomposition in the context of radial neural networks, and (3) precipitates an algorithm to reduce the widths of the hidden layers in such networks.

Original: Our main results, namely Theorems 5.3 and 5.7, may potentially generalize in several ways.
Revised: Our main results, namely Theorems 4.3 and 4.7, may potentially generalize in several ways.

Original: First, these theorems are only meaningful if n i > n i − 1 + 1 for some i (otherwise n red = n ).
Revised: First, these theorems are only meaningful if n i > n i − 1 + 1 for some i (otherwise n red = n ).

Original: Many networks such as decoders, super-resolution mappings, and GANs fulﬁll this criteria, but others do not.
Revised: Many networks such as decoders, super-resolution mappings, and GANs fulﬁll this criteria, but others do not.

Original: We expect our techniques to prove useful in achieving meaningful width reductions in other cases.
Revised: Our techniques may achieve meaningful width reductions in other cases.

Original: Second, the hypothesis that the activation functions are radial is crucial for our results, as they commute with orthogonal matrices, and hence interact favorably with QR decompositions.
Revised: Second, the hypothesis that the activation functions are radial is crucial for our results, as they commute with orthogonal matrices, and hence interact favorably with QR decompositions.

Original: However, similar dimensional-reduction procedures may well be possible for other activation functions which relate to other matrix decompositions.
Revised: However, similar dimensional-reduction procedures may well be possible for other activation functions which relate to other matrix decompositions.

Revised: In particular, the bottleneck effect discovered by Serra et al.

Revised: (2018) for ReLU networks bears a striking similarity to our ascending condition.

Revised: Third, though Theorem 4.7 does not hold for coordinate-dependent optimizers like Adam, other optimizers in the same spirit may be compatible.

Original: Third, there may be enhancements of our results that incorporate regularization and assert provable improvements to generalization.
Revised: Fourth, there may be enhancements of our results that incorporate regularization and assert provable improvements to generalization.

Original: For example, the loss function of a radial neural network with dimension vector n remains invariant for the O ( n ) action after adding an L 2 regularization term.
Revised: For example, the loss function of a radial neural network with dimension vector n remains invariant for the O ( n ) action after adding an L 2 regularization term.

Original: Finally, the conceptual ﬂexibility of quiver representation theory can encapsulate neural networks beyond MLPs, including convolutional neural networks (CNNs), equivariant neural networks, recurrent neural networks, graph neural networks, and others.
Revised: Finally, the conceptual ﬂexibility of quiver representation theory can encapsulate neural networks beyond MLPs, including CNNs, equivariant neural networks, RNNs, GNNs, and others.

Original: Our result in Theorem 5.7 relates projected gradient descent γ proj for the original network to gradient descent γ for the reduced network.
Revised: Our result in Theorem 4.7 relates projected gradient descent γ proj for the original network to gradient descent γ for the reduced network.

Original: In order to make theoretical conclusions about optimization of the original versus the reduced network, it is necessary to relate gradient descent on both spaces.
Revised: In order to make theoretical conclusions about optimization of the original versus the reduced network, it is necessary to relate gradient descent on both spaces.

Original: To realize this, we conjecture, based on preliminary experimental evidence, that our version of projected gradient descent is a ﬁrst-order approximation of usual gradient descent; we are pursuing work to make this relationship precise.
Revised: To realize this, we conjecture, based on preliminary experimental evidence, that our version of projected gradient descent is a ﬁrst-order approximation of usual gradient descent; we are pursuing work to make this relationship precise.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Our work is primarily focused on theoretical foundations of machine learning, however, it does have a direct application in the form a model compression.
Revised: Our work is primarily focused on theoretical foundations of machine learning, however, it does have a direct application in the form a model compression.

Original: Model compression is largely beneﬁcial to the world since it allows for inference to run on smaller systems which use less energy.
Revised: Model compression is largely beneﬁcial to the world since it allows for inference to run on smaller systems which use less energy.

Original: On the other hand, when models may be run on smaller systems such as smartphones, it is easier to use deep models covertly, for example, for facial recognition and surveillance.
Revised: On the other hand, when models may be run on smaller systems such as smartphones, it is easier to use deep models covertly, for example, for facial recognition and surveillance.

Original: This may make abuses of deep learning technology easier to hide.
Revised: This may make abuses of deep learning technology easier to hide.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The theoretical results of this paper, namely Theorem 5.3 and Theorem 5.
Revised: The theoretical results of this paper, namely Theorem 4.3 and Theorem 4.

Original: may be independently veriﬁed through their proofs, which we include in their entirety in Appendices E and F, including all necessary deﬁnitions, lemmas and hypotheses in precise and complete mathematical language.
Revised: may be independently veriﬁed through their proofs, which we include in their entirety in Appendices E and F, including all necessary deﬁnitions, lemmas and hypotheses in precise and complete mathematical language.

Original: The empirical veriﬁcation of Section 6 may be reproduced using the code included with the supplementary materials.
Revised: The empirical veriﬁcation of Section 5 may be reproduced using the code included with the supplementary materials.

Original: In addition, Algorithm 1 is written in detailed pseudocode, allowing readers to recreate our algorithm in a programming language of their choosing
Revised: In addition, Algorithm 1 is written in detailed pseudocode, allowing readers to recreate our algorithm in a programming language of their choosing

Original: A O RGANIZATION OF APPENDICES
Revised: A O RGANIZATION OF APPENDICES

Original: This paper is a contribution to the mathematical foundations of machine learning, and our results are motivated by expanding the applicability and performance of neural networks.
Revised: This paper is a contribution to the mathematical foundations of machine learning, and our results are motivated by expanding the applicability and performance of neural networks.

Original: At the same time, we use precise mathematical formalism throughout, and emphasize the rich abstract structures underlying neural networks.
Revised: At the same time, we use precise mathematical formalism throughout, and emphasize the rich abstract structures underlying neural networks.

Original: The purposes of these appendices are several:
Revised: The purposes of these appendices are several:

Original: To clarify the mathematical conventions and terminology, thus making the paper more accessible. 2.
Revised: To clarify the mathematical conventions and terminology, thus making the paper more accessible. 2.

Original: To provide full proofs of the main results.
Revised: To provide full proofs of the main results.

Original: To develop context around various constructions appearing in the main text.
Revised: To develop context around various constructions appearing in the main text.

Original: To discuss in detail examples, special cases, and generalizations of our results.
Revised: To discuss in detail examples, special cases, and generalizations of our results.

Original: We now give a summary of the contents of the appendices.
Revised: We now give a summary of the contents of the appendices.

Original: Appendix B consists of preliminary material.
Revised: Appendix B consists of preliminary material.

Original: We ﬁrst review the necessary background on linear algebra and group theory, and then state the version of the QR decomposition that is relevant for this paper.
Revised: We ﬁrst review the necessary background on linear algebra and group theory, and then state the version of the QR decomposition that is relevant for this paper.

Original: Appendix C serves to establish notation used in the context of gradient descent, and to prove an interaction between gradient descent and orthogonal transformations (Proposition C.2).
Revised: Appendix C serves to establish notation used in the context of gradient descent, and to prove an interaction between gradient descent and orthogonal transformations (Proposition C.2).

Original: Appendix D supplements 4.
Revised: Appendix D supplements 3.

Original: Speciﬁcally, in Appendix D, we provide expanded foundations for our proposed framework of neural networks as quiver representations.
Revised: Speciﬁcally, in Appendix D, we provide expanded foundations for our proposed framework of neural networks as quiver representations.

Original: In particular, we give a proof of Lemma 4.1, explain symmetries on the space of all neural networks with a speciﬁed dimension vector, and prove equivariance results.
Revised: In particular, we give a proof of Lemma 3.1, explain symmetries on the space of all neural networks with a speciﬁed dimension vector, and prove equivariance results.

Original: Finally, we explain how the “non-negative homogeneity” property of pointwise ReLU activation functions is a special case of our results (c.f Proposition D. and Remark D.7).
Revised: Finally, we explain how the “non-negative homogeneity” property of pointwise ReLU activation functions is a special case of our results (c.f Proposition D.12 and Remark D.14).

Original: In Appendix E, we ﬁrst establish elementary results related to radial functions and radial neural networks in Section E.1, and then give a proof of the QR Decomposition result (Theorem 5.3) in
Revised: In Appendix E, we ﬁrst establish elementary results related to radial functions and radial neural networks in Section E.1, and then give a proof of the QR Decomposition result (Theorem 4.3) in

Original: Section E.2.
Revised: Section E.2.

Original: The proof relies on Algorithm 1.
Revised: The proof relies on Algorithm 1.

Original: In Appendix F, we focus on projected gradient descent.
Revised: In Appendix F, we focus on projected gradient descent.

Original: Speciﬁcally, we ﬁrst introduce an ‘interpolating space’ that relates the space of representations with a given dimension vector to the space of representations with the reduced dimension vector (Section F.1).
Revised: Speciﬁcally, we ﬁrst introduce an ‘interpolating space’ that relates the space of representations with a given dimension vector to the space of representations with the reduced dimension vector (Section F.1).

Original: Using the interpolating space, we deﬁne a projected version of gradient descent (Section F.2) and give a proof of Theorem 5.7, which relates projected gradient descent and the QR decomposition for radial neural networks.
Revised: Using the interpolating space, we deﬁne a projected version of gradient descent (Section F.2) and give a proof of Theorem 4.7, which relates projected gradient descent and the QR decomposition for radial neural networks.

Original: We devote Appendices G and H to special cases and generalizations of the constructions and results of the main text.
Revised: We devote Appendices G and H to special cases and generalizations of the constructions and results of the main text.

Original: Speciﬁcally, we consider a version of our results for neural networks with no biases (Section G.1), a generalization involving shifts in radial functions (Section G.2), and a formulation and generalization of our results using the language of category theory (Appendix H).
Revised: Speciﬁcally, we consider a version of our results for neural networks with no biases (Section G.1), a generalization involving shifts in radial functions (Section G.2), and a formulation and generalization of our results using the language of category theory (Appendix H).

Original: B L INEAR ALGEBRA
Revised: B L INEAR ALGEBRA

Original: In this appendix, we ﬁrst review basic notation and concepts from linear algebra, group theory, and representation theory.
Revised: In this appendix, we ﬁrst review basic notation and concepts from linear algebra, group theory, and representation theory.

Original: We then state a version of the QR decomposition.
Revised: We then state a version of the QR decomposition.

Original: References include Petersen et al.
Revised: References include Petersen et al.

Original: (2012), Quarteroni et al.
Revised: (2012), Quarteroni et al.

Original: (2007), and Dummit & Foote (2003).
Revised: (2007), and Dummit & Foote (2003).

Original: This appendix supplements the ﬁrst part of Section 3 in the main text.
Revised: This appendix supplements the ﬁrst part of Section ?? in the main text.

Original: B. B ASICS
Revised: B. B ASICS

Original: For positive integers m and n , let Hom( R m , R n ) or R m × n denote the vector space of n by m matrices with entries in R , that is, the vector space of linear functions (also known as homomorphisms ) from R m to R n .
Revised: For positive integers m and n , let Hom( R m , R n ) or R m × n denote the vector space of n by m matrices with entries in R , that is, the vector space of linear functions (also known as homomorphisms ) from R m to R n .

Original: The general linear group GL n ( R ) consists of the set of invertible n by n matrices, with the operation of matrix multiplication.
Revised: The general linear group GL n ( R ) consists of the set of invertible n by n matrices, with the operation of matrix multiplication.

Original: Equivalently, GL n ( R ) consists of all linear automorphisms of R n , with the operation of composition.
Revised: Equivalently, GL n ( R ) consists of all linear automorphisms of R n , with the operation of composition.

Original: Such automorphisms are given precisely by change-of-basis transformations.
Revised: Such automorphisms are given precisely by change-of-basis transformations.

Original: The unit is the identity n by n matrix, denoted id n .
Revised: The unit is the identity n by n matrix, denoted id n .

Original: The orthogonal group O ( n ) is the subgroup of GL n ( R ) consisting of all matrices Q such that Q T Q = id n .
Revised: The orthogonal group O ( n ) is the subgroup of GL n ( R ) consisting of all matrices Q such that Q T Q = id n .

Original: Such a matrix is called an orthogonal transformation .
Revised: Such a matrix is called an orthogonal transformation .

Original: Let G be a group.
Revised: Let G be a group.

Original: An action (or representation ) of G on the vector space R n is the data of an invertible n by n matrix ρ ( g ) for every group element g ∈ G , such that (1) for all g, h ∈ G , the matrix ρ ( gh ) is the product of the matrices ρ ( g ) and ρ ( h ) , and (2) for the identity element 1 G ∈ G , we have ρ (1 G ) =
Revised: An action (or representation ) of G on the vector space R n is the data of an invertible n by n matrix ρ ( g ) for every group element g ∈ G , such that (1) for all g, h ∈ G , the matrix ρ ( gh ) is the product of the matrices ρ ( g ) and ρ ( h ) , and (2) for the identity element 1 G ∈ G , we have ρ (1 G ) =

Original: id n .
Revised: id n .

Original: In other words, an action amounts to a group homomorphism ρ : G → GL n ( R ) .
Revised: In other words, an action amounts to a group homomorphism ρ : G → GL n ( R ) .

Original: We often abbreviate ρ ( g )( v ) as simply g · v , for g ∈ G and v ∈ R n .
Revised: We often abbreviate ρ ( g )( v ) as simply g · v , for g ∈ G and v ∈ R n .

Original: A function f : R n → R (non-linear in general) is invariant for the action of G if f ( g · v ) =
Revised: A function f : R n → R (non-linear in general) is invariant for the action of G if f ( g · v ) =

Original: f ( v ) for all g ∈ G and v ∈ R n .
Revised: f ( v ) for all g ∈ G and v ∈ R n .

Original: Suppose m ≤ n .
Revised: Suppose m ≤ n .

Original: Then inc m,n : R m (cid:44) → R n denotes the standard inclusion into the ﬁrst m coordinates: ( v 1 , . .
Revised: Then inc m,n : R m (cid:44) → R n denotes the standard inclusion into the ﬁrst m coordinates: ( v 1 , . .

Original: . , v m ) (cid:55)→ ( v 1 , . . .
Revised: . , v m ) (cid:55)→ ( v 1 , . . .

Original: , v m , 0 , . .
Revised: , v m , 0 , . .

Original: The standard projection π n,m : R n (cid:16) R m onto the ﬁrst m coordinates is deﬁned as: ( v 1 , . . .
Revised: The standard projection π n,m : R n (cid:16) R m onto the ﬁrst m coordinates is deﬁned as: ( v 1 , . . .

Original: , v m , v m +1 , . .
Revised: , v m , v m +1 , . .

Original: . , v n ) (cid:55)→ ( v 1 , . . .
Revised: . , v n ) (cid:55)→ ( v 1 , . . .

Original: , v m )
Revised: , v m )

Original: An afﬁne map from R m to R n is one of the form x
Revised: An afﬁne map from R m to R n is one of the form x

Original: (cid:55)→ Ax + b for some n × m matrix A and n -dimensional vector b .
Revised: (cid:55)→ Ax + b for some n × m matrix A and n -dimensional vector b .

Original: The set of afﬁne maps from R m to R n can be identiﬁed with the vector space
Revised: The set of afﬁne maps from R m to R n can be identiﬁed with the vector space

Original: Hom( R m +1 , R n ) , as we now explain.
Revised: Hom( R m +1 , R n ) , as we now explain.

Original: First, given A and b as above, we form the n × ( m + 1) matrix
Revised: First, given A and b as above, we form the n × ( m + 1) matrix

Original: [ b A ] .
Revised: [ b A ] .

Original: Conversely, the afﬁne map corresponding to
Revised: Conversely, the afﬁne map corresponding to

Original: W ∈ Hom( R m +1 , R n ) is given by
Revised: W ∈ Hom( R m +1 , R n ) is given by

Original: [Equation]
Revised: [Equation]

Original: where ext m is deﬁned as follows:
Revised: where ext m is deﬁned as follows:

Original: Deﬁnition B.1.
Revised: Deﬁnition B.1.

Original: For m ≥ 1 , let ext m : R m →
Revised: For m ≥ 1 , let ext m : R m →

Original: R m + be the map taking v =
Revised: R m + be the map taking v =

Original: ( v 1 , v 2 , . .
Revised: ( v 1 , v 2 , . .

Original: . , v m ) ∈ R m to (1 , v 1 , v 2 , . .
Revised: . , v m ) ∈ R m to (1 , v 1 , v 2 , . .

Original: . , v m
Revised: . , v m

Original: ) ∈
Revised: ) ∈

Original: R m +1 .
Revised: R m +1 .

Original: B. T HE QR DECOMPOSITION
Revised: B. T HE QR DECOMPOSITION

Original: In this section, we recall the QR decomposition and note several relevant facts.
Revised: In this section, we recall the QR decomposition and note several relevant facts.

Original: For integers n and m , let Hom upper ( R m , R n ) denote the vector space of upper triangular n by m matrices.
Revised: For integers n and m , let Hom upper ( R m , R n ) denote the vector space of upper triangular n by m matrices.

Original: Theorem B.2 (QR Decomposition) .
Revised: Theorem B.2 (QR Decomposition) .

Original: The following map is surjective:
Revised: The following map is surjective:

Original: [Equation]
Revised: [Equation]

Original: In other words, any matrix can be written as the product of an orthogonal matrix and an uppertriangular matrix.
Revised: In other words, any matrix can be written as the product of an orthogonal matrix and an uppertriangular matrix.

Original: When m ≤ n , the last n − m rows of any matrix in Hom upper ( R m , R n ) are zero, and the top m rows form an upper-triangular m by m matrix.
Revised: When m ≤ n , the last n − m rows of any matrix in Hom upper ( R m , R n ) are zero, and the top m rows form an upper-triangular m by m matrix.

Original: These observations lead to the following special case of the QR decomposition:
Revised: These observations lead to the following special case of the QR decomposition:

Original: Corollary B.3.
Revised: Corollary B.3.

Original: Suppose m ≤ n .
Revised: Suppose m ≤ n .

Original: The following map is surjective:
Revised: The following map is surjective:

Original: [Equation]
Revised: [Equation]

Original: We make some remarks:
Revised: We make some remarks:

Original: There are several algorithms for computing the QR decomposition of a given matrix.
Revised: There are several algorithms for computing the QR decomposition of a given matrix.

Original: One is Gram–Schmidt orthogonalization, and another is the method of Householder reﬂections.
Revised: One is Gram–Schmidt orthogonalization, and another is the method of Householder reﬂections.

Original: The latter has computational complexity O ( n 2 m ) in the case of a n × m matrix with n ≥ m .
Revised: The latter has computational complexity O ( n 2 m ) in the case of a n × m matrix with n ≥ m .

Original: The package numpy includes a function numpy.linalg.qr that computes the QR decomposition of a matrix using Householder reﬂections.
Revised: The package numpy includes a function numpy.linalg.qr that computes the QR decomposition of a matrix using Householder reﬂections.

Original: The QR decomposition is not unique in general, or, in other words
Revised: The QR decomposition is not unique in general, or, in other words

Original: , the map µ is not injective in general.
Revised: , the map µ is not injective in general.

Original: For example, if n > m , each ﬁber of µ contains a copy of O ( n − m ) .
Revised: For example, if n > m , each ﬁber of µ contains a copy of O ( n − m ) .

Original: The QR decomposition is unique (in a certain sense) for invertible square matrices.
Revised: The QR decomposition is unique (in a certain sense) for invertible square matrices.

Original: To be precise, let B + n be the subset of of Hom upper ( R n , R n ) consisting of upper triangular matrices with positive entries along the diagonal.
Revised: To be precise, let B + n be the subset of of Hom upper ( R n , R n ) consisting of upper triangular matrices with positive entries along the diagonal.

Original: If n = m , then both B + n and O ( n ) are subgroups of GL n ( R ) , and the multiplication map O ( n ) × B + n → GL n ( R ) is bijective.
Revised: If n = m , then both B + n and O ( n ) are subgroups of GL n ( R ) , and the multiplication map O ( n ) × B + n → GL n ( R ) is bijective.

Original: However, the QR decomposition is not unique for non-invertible square matrices.
Revised: However, the QR decomposition is not unique for non-invertible square matrices.

Original: C S YMMETRY AND GRADIENT DESCENT
Revised: C S YMMETRY AND GRADIENT DESCENT

Original: In this appendix, we establish a gradient descent formalism that appears throughout the paper.
Revised: In this appendix, we establish a gradient descent formalism that appears throughout the paper.

Original: Additionally, we explain the interaction between gradient descent and orthogonal transformations, which is a key aspect of our main results.
Revised: Additionally, we explain the interaction between gradient descent and orthogonal transformations, which is a key aspect of our main results.

Original: This appendix supplements the last part of Section 3.
Revised: This appendix supplements the last part of Section ?? .

Original: Fix a positive integer p .
Revised: Fix a positive integer p .

Original: Let V
Revised: Let V

Original: = R p and L : V → R be a differentiable function 4 .
Revised: = R p and L : V → R be a differentiable function 4 .

Original: Semantically, p is the number of independent parameters, V is the parameter space, and L is the loss function associated to a batch of training data.
Revised: Semantically, p is the number of independent parameters, V is the parameter space, and L is the loss function associated to a batch of training data.

Original: We write elements of V as tuples, e.g. v = ( v 1 , . .
Revised: We write elements of V as tuples, e.g. v = ( v 1 , . .

Original: . , v p ) , or as column vectors.
Revised: . , v p ) , or as column vectors.

Original: We write elements of the dual vector space V ∗ as row vectors.
Revised: We write elements of the dual vector space V ∗ as row vectors.

Original: The differential and gradient of L are deﬁned as:
Revised: The differential and gradient of L are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Remark C.1.
Revised: Remark C.1.

Original: Suppose that L is in fact a linear map.
Revised: Suppose that L is in fact a linear map.

Original: Then L belongs to V ∗ and d L is the constant map taking each v ∈ V to L .
Revised: Then L belongs to V ∗ and d L is the constant map taking each v ∈ V to L .

Original: That is, d v L =
Revised: That is, d v L =

Original: L as maps V → R .
Revised: L as maps V → R .

Original: A step of gradient descent with respect to L at learning rate η > 0 is the function
Revised: A step of gradient descent with respect to L at learning rate η > 0 is the function

Original: [Equation]
Revised: [Equation]

Original: Hence, gradient descent updates the i -th coordinate v i of v to v i − η · ∂ L ∂x i | v , for i = 1 , . . .
Revised: Hence, gradient descent updates the i -th coordinate v i of v to v i − η · ∂ L ∂x i | v , for i = 1 , . . .

Original: , p .
Revised: , p .

Original: When η is clear from context, we abbreviate γ η by γ .
Revised: When η is clear from context, we abbreviate γ η by γ .

Original: For any k ≥ 0 , the k -fold composition of γ is γ k := γ ◦ γ ◦ · · · ◦ γ , with γ 0 being the identity map on V .
Revised: For any k ≥ 0 , the k -fold composition of γ is γ k := γ ◦ γ ◦ · · · ◦ γ , with γ 0 being the identity map on V .

Original: Throughout the appendices, all proofs involving gradient descent reduce to the η = 1 case.
Revised: Throughout the appendices, all proofs involving gradient descent reduce to the η = 1 case.

Original: The arguments for general η > 0 and can be obtained either by inserting the scalar η where necessary, or by rescaling the function L .
Revised: The arguments for general η > 0 and can be obtained either by inserting the scalar η where necessary, or by rescaling the function L .

Original: Proposition C.2.
Revised: Proposition C.2.

Original: Let Q be an orthogonal transformation of V =
Revised: Let Q be an orthogonal transformation of V =

Original: R p such that L ◦ Q = L .
Revised: R p such that L ◦ Q = L .

Original: Then, for any v ∈ V and k ≥ 0 , we have:
Revised: Then, for any v ∈ V and k ≥ 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We give a proof in the case η = 1 .
Revised: We give a proof in the case η = 1 .

Original: The general case follows similarly.
Revised: The general case follows similarly.

Original: The hypothesis
Revised: The hypothesis

Original: L ◦ Q = L implies that:
Revised: L ◦ Q = L implies that:

Original: [Equation]
Revised: [Equation]

Original: where d Q ( v ) Q = Q since Q is a linear map.
Revised: where d Q ( v ) Q = Q since Q is a linear map.

Original: The fact that Q is an orthogonal transformation implies that Q − 1 = Q T .
Revised: The fact that Q is an orthogonal transformation implies that Q − 1 = Q T .

Original: The ﬁrst claim follows.
Revised: The ﬁrst claim follows.

Original: For the second claim, we ﬁrst compute:
Revised: For the second claim, we ﬁrst compute:

Original: [Equation]
Revised: [Equation]

Original: Inductive reasoning shows that γ k ( Q ( v ))
Revised: Inductive reasoning shows that γ k ( Q ( v ))

Original: =
Revised: =

Original: Q ( γ k ( v )) for all k ≥ 0 .
Revised: Q ( γ k ( v )) for all k ≥ 0 .

Original: The following two results are not necessary for the results of the current paper, but may be of interest to readers.
Revised: The following two results are not necessary for the results of the current paper, but may be of interest to readers.

Original: We omit the proofs (which are straightforward).
Revised: We omit the proofs (which are straightforward).

Original: Proposition C.3.
Revised: Proposition C.3.

Original: Suppose that a group G acts on V by orthogonal transformations and that L is
Revised: Suppose that a group G acts on V by orthogonal transformations and that L is

Original: G -invariant.
Revised: G -invariant.

Original: Then there is a well-deﬁned gradient descent map on the set of G -orbits on V : (cid:101) γ : V/G →V/G.
Revised: Then there is a well-deﬁned gradient descent map on the set of G -orbits on V : [Equation] /G →

Revised: [Equation]

Revised: /G.

Original: Proposition C.4.
Revised: Proposition C.4.

Original: Let M be a linear endomorphism of V
Revised: Let M be a linear endomorphism of V

Original: =
Revised: =

Original: R p such that L ◦ M = L .
Revised: R p such that L ◦ M = L .

Original: For any v ∈ V , we have: ∇ v L = M T ( ∇ M ( v ) L ) .
Revised: For any v ∈ V , we have: ∇ v L = M T ( ∇ M ( v ) L ) .

Original: If M = P is an orthogonal projection, i
Revised: If M = P is an orthogonal projection, i

Original: e. P 2 = P = P T , then, for any v ∈ V and k ≥ 0 , we have:
Revised: e. P 2 = P = P T , then, for any v ∈ V and k ≥ 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: D T HE NEURAL QUIVER AND NEURAL NETWORKS
Revised: D Q UIVERS AND NEURAL NETWORKS

Original: In this appendix, we expand on Section 4 of the main text.
Revised: In this appendix, we expand on Section 3 of the main text.

Original: We explain in more detail the relation between representations of the neural quiver and afﬁne maps, describe an action of GL hidden n on the space of neural networks Neur ( Q L , n ) , and prove equivariance results related to this action.
Revised: We explain in more detail the relation between representations of the neural quiver and afﬁne maps, describe an action of GL hidden n on the space of neural networks Neur ( Q L , n ) , and prove equivariance results related to this action.

Original: References for general quiver representation theory include (Nakajima et al., 1998) and (Kirillov Jr, 2016).
Revised: References for general quiver representation theory include (Nakajima et al., 1998) and (Kirillov Jr, 2016).

Original: D. T HE NEURAL QUIVER
Revised: D. N EURAL - ADAPTED QUIVERS AND Q - NEURAL NETWORKS

Revised: Deﬁnition D.1. A neural-adapted quiver is a quiver Q = ( I, E ) together with a partition I =

Revised: I in ∪ B ∪ H ∪ { i out } of the vertex set such that the following hold:

Revised: The quiver Q is acyclic and no two edges share the same source and target.

Revised: The vertex i out is the unique sink of Q .

Revised: This is the output vertex.

Revised: The set I in is non-empty, and every vertex therein is a source of Q .

Revised: These are the input vertices.

Revised: Every vertex in B is a source of Q .

Revised: These are the bias vertices.

Revised: Every vertex in H is neither a source nor a sink, and admits a path from an input vertex.

Revised: These are the hidden vertices.

Revised: Given a quiver satisfying condition (1) above, and with a

Revised: unique

Revised: sink, one can partition its vertex set in order to give it the structure of a neural-adapted quiver.

Revised: This partition is generally not unique.

Revised: Example D.2.

Revised: We list examples of neural-adapted quivers:

Revised: The neural quiver with ﬁve layers: • (cid:47) (cid:47) • (cid:47) (cid:47) • (cid:47) (cid:47) • (cid:47) (cid:47) • (cid:47) (cid:47) • • (cid:55) (cid:55) (cid:63) (cid:63) (cid:79) (cid:79) (cid:95) (cid:95) (cid:103) (cid:103) One can see that there is a unique neural-adapted structure on this quiver.

Revised: Namely, the unique sink is the output vertex, the source on the far left is the sole input vertex, and the source on the bottom is the sole bias vertex.

Revised: All other vertices are hidden vertices, and each admits a unique path from the input vertex.

Revised: The no-bias neural quiver with ﬁve layers: • (cid:47) (cid:47) • (cid:47) (cid:47) • (cid:47) (cid:47) • (cid:47) (cid:47) • (cid:47) (cid:47) • There is a unique neural-adapted

Revised: structure

Revised: on this

Revised: quiver

Revised: , and it has the property that the set of bias vertices is empty.

Revised: [Table]

Revised: Once again, there is a unique neural-adapted structure on this quiver.

Revised: Given a neural adapted quiver, one exclusively considers dimension vectors whose value at the bias vertices is equal to 1 .

Revised: Thus, a dimension vector for a neural-adapted quiver amounts to a tuple of nonnegative integers indexed by the vertices in I \ B =

Revised: I in ∪ H ∪{ i out } .

Revised: We abbreviate d in := (cid:80) i ∈ I in d i and d out :

Revised: = d i out .

Revised: We now deﬁne the notion of a Q -neural network, that is, a neural network whose underlying architecture is speciﬁed by a neural-adapted quiver.

Revised: Deﬁnition D.3.

Revised: Let Q be a neural-adapted quiver.

Revised: A Q - neural network consists of the following:

Revised: Hyperparameters.

Revised: A dimension vector d for the quiver Q . 2.

Revised: Trainable parameters.

Revised: A representation W of Q with dimension vector d . 3.

Revised: Activation functions.

Revised: Piece-wise differentiable functions a i : R d i → R d i for i ∈ H ∪ { i out } .

Revised: These are grouped into a tuple a .

Revised: The neural function (also known as the feedforward function) of a Q -neural network is deﬁned recursively, as explained in the following deﬁnition.

Revised: Deﬁnition D.4.

Revised: Let Q be a neural-adapted quiver.

Revised: Let ( W , a ) be a Q -neural network with dimension vector d .

Revised: For every non-bias vertex i ∈ I , deﬁne a function:

Revised: [Equation]

Revised: by setting:

Revised: • If i is an input vertex, f i is the projection map.

Revised: • If i is a bias vertex, f i ≡ 1 is the constant function at 1 .

Revised: • For all other vertices, we use the recursive deﬁnition: f i ( x ) =

Revised: a i ◦ (cid:16)(cid:80)

Revised: j e → i W e f j ( x ) (cid:17)

Revised: The neural function of the neural network ( W , a ) is deﬁned as:

Revised: [Equation]

Revised: Remark D.5.

Revised: Let W be a representation of a neural-adapted quiver Q with dimension vector d .

Revised: For each vertex in H ∪ { i out } , set W i to be the following matrix:

Revised: [Equation]

Revised: where { e 1 , . .

Revised: , e r } = t − 1 ( i ) is the set of edges whose target is i .

Revised: Furthermore, for x ∈ R d in , set f pre i ( x ) to be the following vector:

Revised: [Equation]

Revised:  where j 1 is the source of e 1 , and so forth.

Revised: Then we have that

Revised: [Equation]

Revised: One can order the edges { e 1 , . . .

Revised: , e r } so that all edges emanating from a bias vertex appear before all other edges.

Revised: Note that the size of W i is d i × ( d j 1 + · · · + d j r ) , while the size of f pre i ( x ) is ( d j 1 + · · · + d j r ) × 1 .

Revised: We include the following deﬁnitions for future reference.

Revised: Deﬁnition D.6.

Revised: The parameter symmetry group for representations of a neural-adapted quiver Q with dimension vector d is:

Revised: [Equation]

Revised: Deﬁnition D.7.

Revised: Given a dimension vector d for a neural-adapted quiver Q (which, as always, has value 1 on each bias vertex), one deﬁnes the reduced dimension vector d red as follows.

Revised: First, d red i =

Revised: d i for every i ∈ I in ∪ B .

Revised: Next, for any hidden vertex i , recursively set d red i to be minimum of d i and (cid:80) j ∈

Revised: I

Revised: → i d red j .

Revised: Finally, d redout =

Revised: d out for the output vertex.

Revised: D. T HE NEURAL QUIVER

Original: We begin by recalling the deﬁnition of the neural quiver.
Revised: We begin by recalling the deﬁnition of the neural quiver.

Original: For a positive integer L , the neural quiver
Revised: For a positive integer L , the neural quiver

Original: Q L is the following quiver with L + 2 vertices:
Revised: Q L is the following quiver with L + 2 vertices:

Original: [Equation]
Revised: [Equation]

Original: The  top row contains L + 1 vertices, indexed from i = [-
-] to i =
Revised: The vertices in the top row are indexed from i = [-
-]0 to i =

Original: L .
Revised: L .

Revised: The ﬁrst vertex ( i = 0 ) in the top row is the sole input vertex, the last vertex ( i = L ) is the output vertex, and every other vertex ( 1 ≤ i ≤ L − 1 ) in the top row is a hidden vertex.

Original: Thebottom vertex is the bias vertex, and there isan arrow from the  bias vertex  to each vertex in the top row except for i = 0 .
Revised: The vertex at the bottom is the sole bias vertex and is connected to each vertex in the top row except for i = 0 .

Original: We only consider dimension vectors whose value at the bias vertex is equal to 1 .
Revised: We only consider dimension vectors whose value at the bias vertex is equal to 1 .

Original: Hence, a dimension vector for Q L will refer to an ( L + 1) -tuple of positive integers n = ( n 0 , n 1 , . .
Revised: Hence, a dimension vector for Q L will refer to an ( L + 1) -tuple of positive integers n = ( n 0 , n 1 , . .

Original: . , n L ) .
Revised: . , n L ) .

Original: We now restate and give a full proof of Lemma 4.1:
Revised: We now restate and give a full proof of Lemma 3.1:

Original: Lemma D.1.
Revised: Lemma D.8.

Original: (Lemma 4.1.)
Revised: (Lemma 3.1.)

Original: A representation of Q L with dimension vector n is equivalent to the data of an afﬁne map R n i − 1 → R n i for each i = 1 , . . .
Revised: A representation of Q L with dimension vector n is equivalent to the data of an afﬁne map R n i − 1 → R n i for each i = 1 , . . .

Original: , L .
Revised: , L .

Original: There is an isomorphism of vector spaces:
Revised: There is an isomorphism of vector spaces:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By deﬁnition, a representation of Q L with dimension vector n consists of the data of a matrix
Revised: By deﬁnition, a representation of Q L with dimension vector n consists of the data of a matrix

Original: A i ∈ Hom( R n i − 1 , R n i ) and a vector b i ∈ R n i , for each i = 1 , . . .
Revised: A i ∈ Hom( R n i − 1 , R n i ) and a vector b i ∈ R n i , for each i = 1 , . . .

Original: , L .
Revised: , L .

Original: For each i , the corresponding afﬁne map (cid:102) W i : R n i − 1 → R n i is given by x (cid:55)→
Revised: For each i , the corresponding afﬁne map (cid:102) W i : R n i − 1 → R n i is given by x (cid:55)→

Original: A i x + b i .
Revised: A i x + b i .

Original: Conversely, given the data of an afﬁne map R n i − 1 → R n i , for each i , the linear part deﬁnes an element of Hom( R n i − 1 , R n i ) and the translation part deﬁnes an element of R n i .
Revised: Conversely, given the data of an afﬁne map R n i − 1 → R n i , for each i , the linear part deﬁnes an element of Hom( R n i − 1 , R n i ) and the translation part deﬁnes an element of R n i .

Original: This proves the ﬁrst claim.
Revised: This proves the ﬁrst claim.

Original: The second claim follows from the discussion of afﬁne maps appearing in Section B. Explicitly, given an afﬁne map x
Revised: The second claim follows from the discussion of afﬁne maps appearing in Section B. Explicitly, given an afﬁne map x

Original: (cid:55)→ A i x + b i from R n i − 1 to R n i for each i = 1 , . . .
Revised: (cid:55)→ A i x + b i from R n i − 1 to R n i for each i = 1 , . . .

Original: , L , one deﬁnes an element of (cid:76) [-L i =
-]
Revised: , L , one deﬁnes an element of (cid:76) [-L i =
-]Li =

Original: Hom( R 1+ n i − 1 , R n i ) whose i -th factor is the matrix W i
Revised: Hom( R 1+ n i − 1 , R n i ) whose i -th factor is the matrix W i

Original: =
Revised: =

Original: [ b i A i ] .
Revised: [ b i A i ] .

Original: Conversely, pre-composing a matrix in Hom( R 1+ n i − 1 , R n i ) with the map ext n i − 1 : R n i − 
Revised: Conversely, pre-composing a matrix in Hom( R 1+ n i − 1 , R n i ) with the map ext n i − 1 : R n i − 1 → R 1+ n i − 1 gives an afﬁne map (cid:102) W i from R n i − 1 to R n i . Notation : In light of the preceding lemma, we henceforth denote a representation of Q L as a tuple W =

Revised: ( W i )

Revised: L i =1 , where each W i belongs to Hom( R 1+ n i − 1 , R n i ) .

Original: Recall that the  group GL hidden n is deﬁned as the product of the groups GL n i
Revised: Recall that the parameter symmetry group GL hidden n is deﬁned as the product of the groups GL n i

Original: ( R ) as i runs over the hidden vertices of the neural quiver Q L :GL [Equation] n
Revised: ( R ) as i runs over the hidden vertices of the neural quiver Q L :

Original: [Equation]
Revised: [Equation]

Original: Hence an element of GL hidden n consists of the choice of a transformation of R n i for each hidden vertex i .
Revised: Hence an element of GL hidden n consists of the choice of a transformation of R n i for each hidden vertex i .

Original: There is an action of GL hidden n on Rep ( Q L , n ) given by 5 :
Revised: There is an action of GL hidden n on Rep ( Q L , n ) given by 5 :

Original: [Equation]
Revised: [Equation]

Original: where g =
Revised: where g =

Original: ( g i )
Revised: ( g i )

Original: L − 1 i =1 is an element of GL hidden n , and g 

Revised: L − 1 i =1 is an element of GL hidden n , and g

Original: = id n 0 and g L = id n L .
Revised: = id n 0 and g L = id n L .

Original: D. N EURAL NETWORKS
Revised: D

Revised: N EURAL NETWORKS

Original: We now recall the deﬁnition of a neural network from Section 4.2:
Revised: As a special case of Deﬁnition D.3, we have the following (see also Deﬁnition 3.2 in Section 3.4):

Original: Deﬁnition D.2.
Revised: Deﬁnition D.9.

Original: (Deﬁnition 4.2)
Revised: Let L be a positive integer.

Original: An MLP neural network consists of the following:
Revised: An Q L - neural network consists of the following:

Original: Hyperparameters.
Revised: Hyperparameters.

Original: Apositive integer L and a dimension vector n = ( n 0 , n 1 , n 2 , . .
Revised: A dimension vector n = ( n 0 , n 1 , n 2 , . .

Original: . , n L ) for the neural quiver Q L . 2.
Revised: . , n L ) for the neural quiver Q L . 2.

Original: Trainable parameters.
Revised: Trainable parameters.

Original: A representation W =
Revised: A representation W =

Original: ( W 1 , . . .
Revised: ( W 1 , . . .

Original: , W L ) of the  quiver Q L with dimension vector n .
Revised: , W L ) of the neural quiver Q L with dimension vector n .

Original: So, for i = 1 , . . .
Revised: So, for i = 1 , . . .

Original: , L , we have a matrix W
Revised: , L , we have a matrix W

Original: i ∈
Revised: i ∈

Original: Hom( R 1+ n i − 1 , R n i ) .
Revised: Hom( R 1+ n i − 1 , R n i ) .

Original: Activation functions.
Revised: Activation functions.

Original: Piece-wise differentiable functions a i : R n i → R n i for i = 1 , . . .
Revised: Piece-wise differentiable functions a i : R n i → R n i for i = 1 , . . .

Original: , L .
Revised: , L .

Original: These are grouped into a tuple a =
Revised: These are grouped into a tuple a =

Original: ( a 1 , . .
Revised: ( a 1 , . .

Original: . , a L ) .
Revised: . , a L ) .

Original: As in Section 4.2, we denote by Neur ( Q L , n ) the vector space of neural networks with L layers and dimension vector n , and deﬁne the neural function of a neural network ( W , a ) as:
Revised: As in Section 3.4, we denote by Neur ( Q L , n ) the vector space of neural networks with L layers and dimension vector n , and deﬁne the neural function of a neural network ( W , a ) as:

Original: [Equation]
Revised: [Equation]

Original:  where (cid:102) W i : R n i − 1 → R n i is the afﬁne map corresponding to W i .
Revised: ◦ (cid:102) W 1 ( x ) where (cid:102) W i : R n i − 1 → R n i is the afﬁne map corresponding to W i .

Original: There is an action of the group GL hidden n on Neur ( Q L , n ) given as follows.
Revised: There is an action of the group GL hidden n on Neur ( Q L , n ) given as follows.

Original: The element g =
Revised: The element g =

Original: ( g i )
Revised: ( g i )

Original: L − 1 i =1 ∈ GL hidden n transforms ( W , a )
Revised: L − 1 i =1 ∈ GL hidden n transforms ( W , a )

Original: ∈ Neur ( Q L , n ) as: 
Revised: ∈ Neur ( Q L , n ) as: ( W , a )

Original: [Equation]
Revised: [Equation]

Original: where g 0 = id n 0 and g L = id n L .
Revised: where g 0 = id n 0 and g L = id n L .

Original: The following lemma shows that the neural function is unchanged by the GL hidden n action, that is, if two neural networks are related by the action of an element in
Revised: The following lemma shows that the neural function is unchanged by the GL hidden n action, that is, if two neural networks are related by the action of an element in

Original: GL hidden n , then their neural functions are the same.
Revised: GL hidden n , then their neural functions are the same.

Original: Lemma D.3.
Revised: Lemma D.10.

Original: For any neural network ( W , a ) in Neur ( Q L , n ) and any g in GL hidden n , the neural functions of ( W , a ) and g · ( W , a ) coincide:
Revised: For any neural network ( W , a ) in Neur ( Q L , n ) and any g in GL hidden n , the neural functions of ( W , a ) and g · ( W , a ) coincide:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The element g =
Revised: The element g =

Original: ( g i ) ∈ GL hidden n transforms 
Revised: ( g i ) ∈ GL hidden n transforms (cid:102) W i

Original: [Equation]
Revised: =

Original:  ◦ ext n [Equation] 1 to g
Revised: W i ◦ ext n i − 1 to g

Original: i ◦ W i ◦ ext n i − 1 ◦ g − 1 i −and a i to g
Revised: i ◦ W i ◦ ext n i − 1 ◦ g − 1 i −and a i to g

Original: i ◦ a i ◦ g − 1 i , for i = 1 , . . .
Revised: i ◦ a i ◦ g − 1 i , for i = 1 , . . .

Original: , L (where g 0 = id n 0 and g L = id n L ).
Revised: , L (where g 0 = id n 0 and g L = id n L ).

Original: Thus, g transforms the composition a i ◦ (cid:102) W i to the composition g i ◦ a i ◦ (cid:102) W i ◦ g − 1 i − 1 .
Revised: Thus, g transforms the composition a i ◦ (cid:102) W i to the composition g i ◦ a i ◦ (cid:102) W i ◦ g − 1 i − 1 .

Original: The result now follows from the deﬁnition of the neural function.
Revised: The result now follows from the deﬁnition of the neural function.

Original: We now deﬁne a certain subgroup of GL hidden n that will feature in later discussion.
Revised: We now deﬁne a certain subgroup of GL hidden n that will feature in later discussion.

Original: Deﬁnition D.4.
Revised: Deﬁnition D.11.

Original: The stabilizer in GL hidden n of a choice of activation functions a is deﬁned as:
Revised: The stabilizer in GL hidden n of a choice of activation functions a is deﬁned as:

Original: Z ( a ) = { g = ( g i ) ∈ GL hidden n : g i ◦ a i ◦ g − 1 i = a i for all i = 1 , . . , L − 1 } .
Revised: [Equation]

Original: Note that each a i is a function from R n i to R n i , as is each g i , so the equality g i ◦ a i ◦ g − 1 i = a i is as functions from R n i to R n i .
Revised: Note that each a i is a function from R n i to R n i , as is each g i , so the equality g i ◦ a i ◦ g − 1 i = a i is as functions from R n i to R n i .

Original: As a subgroup of GL hidden n , the stabilizer Z ( a ) acts on Rep ( Q L , n ) .
Revised: As a subgroup of GL hidden n , the stabilizer Z ( a ) acts on Rep ( Q L , n ) .

Original: We now recall from Section 4 the formulation of the optimization problem for neural networks in the language of quiver representations.
Revised: We now recall from Section 3 the formulation of the optimization problem for neural networks in the language of quiver representations.

Original: For the remainder of this section, we ﬁx a dimension vector n for the the neural quiver Q L , and activation functions a = ( a 1 , . . .
Revised: For the remainder of this section, we ﬁx a dimension vector n for the the neural quiver Q L , and activation functions a = ( a 1 , . . .

Original: , a L ) .
Revised: , a L ) .

Original: To a batch of training data X = { ( x j , y j ) } ⊆ R n 0 × R n L , there is an associated loss function on Rep ( Q L , n ) deﬁned asL = L X : Rep [Equation] Q L , n )
Revised: To a batch of training data X = { ( x j , y j ) } ⊆ R n 0 × R n L , there is an associated loss function on Rep ( Q L , n ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where C : R n L × R n L → R is a cost function on R n L .
Revised: where C : R n L × R n L → R is a cost function on R n L .

Original: Following the formalism of Section 3 (or
Revised: Following the formalism of Section ?? (or

Original: Appendix C), we obtain a gradient descent map γ : Rep ( Q L , n ) → Rep ( Q L , n ) for any learning rate η > 0 .Hence:
Revised: Appendix C), we obtain a gradient descent map [Equation] for any learning rate η > 0 .

Original:  → Upshot : the backpropogation algorithm can be regarded as taking place on the vector space Rep ( Q L , n ) of representations of the neural quiver Q L .
Revised: Hence: → Upshot : the backpropogation algorithm can be regarded as taking place on the vector space Rep ( Q L , n ) of representations of the neural quiver Q L .

Original: To conclude this section, we demonstrate that the loss function is invariant for the action of the group
Revised: To conclude this section, we demonstrate that the loss function is invariant for the action of the group

Original: Z ( a ) .
Revised: Z ( a ) .

Original: Proposition D.5.
Revised: Proposition D.12.

Original: Let a be activation functions, and let L be the loss function associated to a batch of training data.
Revised: Let a be activation functions, and let L be the loss function associated to a batch of training data.

Original: Then, for all g ∈ Z ( a ) and W ∈ Rep ( Q L , n ) , we have:
Revised: Then, for all g ∈ Z ( a ) and W ∈ Rep ( Q L , n ) , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: One ﬁrst veriﬁes using the deﬁnition of Z ( a ) that:
Revised: One ﬁrst veriﬁes using the deﬁnition of Z ( a ) that:

Original: [Equation]
Revised: [Equation]

Original: for any g ∈ Z ( a ) and W ∈ Rep ( Q L , n ) , where the left-hand side invokes the action of Z ( a ) on
Revised: for any g ∈ Z ( a ) and W ∈ Rep ( Q L , n ) , where the left-hand side invokes the action of Z ( a ) on

Original: Rep ( Q L , n ) , while the right-hand side invokes the action of Z ( a ) on Neur ( Q L , n ) .
Revised: Rep ( Q L , n ) , while the right-hand side invokes the action of Z ( a ) on Neur ( Q L , n ) .

Original: Using this fact and Lemma D.3, we see that: f ( g · W , a ) =
Revised: Using this fact and Lemma D.10, we see that: f ( g · W , a ) =

Original: f g · ( W , a ) =
Revised: f g · ( W , a ) =

Original: f ( W , a ) and hence, for any g ∈ Z ( a ) and
Revised: f ( W , a ) and hence, for any g ∈ Z ( a ) and

Original: W ∈ Rep ( Q L , n ) , we have:
Revised: W ∈ Rep ( Q L , n ) , we have:

Original: [Equation]
Revised: [Equation]

Original: Remark D.6.
Revised: Remark D.13.

Original: An alternative proof is as follows.
Revised: An alternative proof is as follows.

Original: The loss function can be written as the composition
Revised: The loss function can be written as the composition

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst map takes W to ( W , a ) (and hence is a section of the natural projection from
Revised: where the ﬁrst map takes W to ( W , a ) (and hence is a section of the natural projection from

Original: Neur ( Q L , n ) to Rep ( Q L , n ) ), and the second map sends ( W , a ) to the sum (cid:80) j C ( f ( W , a ) ( x j ) , y j ) .
Revised: Neur ( Q L , n ) to Rep ( Q L , n ) ), and the second map sends ( W , a ) to the sum (cid:80) j C ( f ( W , a ) ( x j ) , y j ) .

Original: One veriﬁes directly that s a is equivariant for the action of the group
Revised: One veriﬁes directly that s a is equivariant for the action of the group

Original: Z ( a ) , while the second map is invariant for GL hidden n , of which Z ( a ) is a subgroup.
Revised: Z ( a ) , while the second map is invariant for GL hidden n , of which Z ( a ) is a subgroup.

Original: Remark D.7.
Revised: Remark D.14.

Original: Suppose a i : R n i → R n i is pointwise ReLU for each i .
Revised: Suppose a i : R n i → R n i is pointwise ReLU for each i .

Original: Then Z ( a ) consists of those g
Revised: Then Z ( a ) consists of those g

Original: = ( g i ) ∈ GL hidden n such that each g i ∈ GL n i ( R ) is a diagonal matrix with positive entries along the diagonal.
Revised: = ( g i ) ∈ GL hidden n such that each g i ∈ GL n i ( R ) is a diagonal matrix with positive entries along the diagonal.

Original: In symbols:
Revised: In symbols:

Original: [Equation]
Revised: [Equation]

Original: Hence, the “non-negative homogeneity” (or “positive scaling invariance”) property of pointwise
Revised: Hence, the “non-negative homogeneity” (or “positive scaling invariance”) property of pointwise

Original: ReLU activation functions appearing in Dinh et al.
Revised: ReLU activation functions appearing in Dinh et al.

Original: ; Neyshaburet al.
Revised: ; Neyshabur

Original: ; Meng et al.

Original: (2019) emerges as a special case of Proposition D.5.

Original: E R ADIAL NEURAL NETWORKS AND THE QR DECOMPOSITION
Revised: E R ADIAL NEURAL NETWORKS AND THE QR DECOMPOSITION

Original: In this appendix, we give a proof of the main theoretical result (Theorem 5.3) involving a QR decomposition for radial neural networks.
Revised: In this appendix, we give a proof of the main theoretical result (Theorem 4.3) involving a QR decomposition for radial neural networks.

Original: To this end, we ﬁrst collect results about radial functions and radial neural networks.
Revised: To this end, we ﬁrst collect results about radial functions and radial neural networks.

Original: E. R ADIAL FUNCTIONS
Revised: E. R ADIAL FUNCTIONS

Original: We ﬁrst recall the deﬁnition of radial functions from Section 5.1 above.
Revised: We ﬁrst recall the deﬁnition of radial functions from Section 4.1 above.

Original: Let h : R → R be a piece-wise differentiable function such that lim r
Revised: Let h : R → R be a piece-wise differentiable function such that lim r

Original: → 0 h ( r ) r exists.
Revised: → 0 h ( r ) r exists.

Original: For any n ≥ 1 , set:
Revised: For any n ≥ 1 , set:

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: R n → R n ;
Revised: R n → R n ;

Original: [Equation]
Revised: [Equation]

Original: A function a : R n → R n is called radial if a = h ( n ) for some h : R → R .
Revised: A function a : R n → R n is called radial if a = h ( n ) for some h : R → R .

Original: Lemma E.1.
Revised: Lemma E.1.

Original: Let a = h ( n ) : R n → R n be a radial function on R n .
Revised: Let a = h ( n ) : R n → R n be a radial function on R n .

Original: The radial function a commutes with any orthogonal transformation of R n .
Revised: The radial function a commutes with any orthogonal transformation of R n .

Original: In symbols: a ◦ Q = Q ◦ a for any Q ∈ O ( n ) .
Revised: In symbols: a ◦ Q = Q ◦ a for any Q ∈ O ( n ) .

Original: If m ≤ n and inc m,n : R m (cid:44) → R n is the standard inclusion, then:
Revised: If m ≤ n and inc m,n : R m (cid:44) → R n is the standard inclusion, then:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Suppose Q ∈ O ( n ) is an orthogonal transformation of R n .
Revised: Suppose Q ∈ O ( n ) is an orthogonal transformation of R n .

Original: Since Q is norm-preserving, we have | Qv | = | v | for any v ∈ R n .
Revised: Since Q is norm-preserving, we have | Qv | = | v | for any v ∈ R n .

Original: Since Q is linear, we have Q ( λv ) =
Revised: Since Q is linear, we have Q ( λv ) =

Original: λQv for any λ ∈ R and v ∈ R n .
Revised: λQv for any λ ∈ R and v ∈ R n .

Original: Using the deﬁnition of a = h ( n ) we compute:
Revised: Using the deﬁnition of a = h ( n ) we compute:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst claim follows.
Revised: The ﬁrst claim follows.

Original: The second claim is an elementary veriﬁcation.
Revised: The second claim is an elementary veriﬁcation.

Original: Remark E.2.
Revised: Remark E.2.

Original: More generally, it is straightforward to show that the restriction of the radial function a to a linear subspace of R n is a radial function on that subspace.
Revised: More generally, it is straightforward to show that the restriction of the radial function a to a linear subspace of R n is a radial function on that subspace.

Original: Next, we recall the deﬁnition of a radial neural network (as stated in Section 5.1).
Revised: Next, we recall the deﬁnition of a radial neural network (as stated in Section 4.1).

Original: Let L be a positive integer and n be a dimension vector for the neural quiver Q L .
Revised: Let L be a positive integer and n be a dimension vector for the neural quiver Q L .

Original: A neural network ( W , a ) ∈
Revised: A neural network ( W , a ) ∈

Original: Neur ( Q L , n ) is said to be a radial neural network if each activation function a i is a radial function on R n i .
Revised: Neur ( Q L , n ) is said to be a radial neural network if each activation function a i is a radial function on R n i .

Original: Let O ( n ) denote the following product of the orthogonal groups:
Revised: Let O ( n ) denote the following product of the orthogonal groups:

Original: [Equation]
Revised: [Equation]

Original: Note that O ( n ) is a subgroup of GL hidden n and hence acts on Rep ( Q L , n ) .
Revised: Note that O ( n ) is a subgroup of GL hidden n and hence acts on Rep ( Q L , n ) .

Original: The following proposition involves the loss function associated to a batch of training data as formulated in Section 4.2.
Revised: The following proposition involves the loss function associated to a batch of training data as formulated in Section 3.4.

Original: Proposition E.3.
Revised: Proposition E.3.

Original: For any radial neural network in Neur ( Q L , n ) and any batch of training data, the loss function L : Rep ( Q L , n ) → R is invariant for the action of O ( n ) .
Revised: For any radial neural network in Neur ( Q L , n ) and any batch of training data, the loss function L : Rep ( Q L , n ) → R is invariant for the action of O ( n ) .

Original: Proof.
Revised: Proof.

Original: Let g
Revised: Let g

Original: = ( g 1 , . .
Revised: = ( g 1 , . .

Original: , g L − 1 ) ∈
Revised: , g L − 1 ) ∈

Original: O ( n ) .
Revised: O ( n ) .

Original: Lemma E.1 implies that the orthogonal transformation g i ∈
Revised: Lemma E.1 implies that the orthogonal transformation g i ∈

Original: O ( n i ) commutes with the radial function a
Revised: O ( n i ) commutes with the radial function a

Original: i = h ( n i ) , for each i , or, equivalently: g i ◦ a i ◦ g − 1 i
Revised: i = h ( n i ) , for each i , or, equivalently: g i ◦ a i ◦ g − 1 i

Original: = a i .
Revised: = a i .

Original: Hence O ( n ) is a subgroup of the centralizer Z ( a ) , and we apply Proposition D.5.
Revised: Hence O ( n ) is a subgroup of the centralizer Z ( a ) , and we apply Proposition D.12.

Original: For clarity, we restate Construction 5.2, which illustrates the ﬂexibility of radial functions in deﬁning the activation functions of a neural network for any given representation of Q L with arbitrary dimension vector.
Revised: For clarity, we restate Construction 4.2, which illustrates the ﬂexibility of radial functions in deﬁning the activation functions of a neural network for any given representation of Q L with arbitrary dimension vector.

Original: Construction E.4.
Revised: Construction E.4.

Original: Suppose h
Revised: Suppose h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: , h L ) is a tuple of functions 6 h i : R → R , and suppose n = ( n 0 , . . .
Revised: , h L ) is a tuple of functions 6 h i : R → R , and suppose n = ( n 0 , . . .

Original: , n L ) is a dimension vector for Q L .
Revised: , n L ) is a dimension vector for Q L .

Original: Consider the tuple of activation functions:
Revised: Consider the tuple of activation functions:

Original: [Equation]
Revised: [Equation]

Original: L ) L (cid:17) .
Revised: L ) L (cid:17) .

Original: Any representation W of Q L with dimension vector n deﬁnes a neural network (cid:16) W , a ( n ) h (cid:17) as in
Revised: Any representation W of Q L with dimension vector n deﬁnes a neural network (cid:16) W , a ( n ) h (cid:17) as in

Original: Deﬁnition 4.2.
Revised: Deﬁnition 3.2.

Original: We write simply ( W , h ) for this network and f ( W , h ) for its neural function.
Revised: We write simply ( W , h ) for this network and f ( W , h ) for its neural function.

Original: exists.
Revised: exists.

Original: E. P ROOF OF T HEOREM 5. 
Revised: E. P ROOF OF T HEOREM 4.

Original: In this section, we give a proof of the QR decomposition appearing in Theorem 5.3.
Revised: In this section, we give a proof of the QR decomposition appearing in Theorem 4.3.

Original: In order to do so, we make some recollections from Section 5.2.
Revised: In order to do so, we make some recollections from Section 4.2.

Original: Given a dimension vector n = ( n 0 , n 1 , . . . , n L ) for the neural quiver Q L , recall that its reduction n red = ( n red0 , n red1 , . . . , n red L ) is deﬁned recursively by setting n red0 = n 0 , then n red i = min( n i , n red i − 1 + 1) for i = 1 , . . .
Revised: Given a dimension vector n = ( n 0 , n 1 , . . . , n L ) for the neural quiver Q L , recall that its reduction n red = ( n red0 , n red1 , . . . , n red L ) is deﬁned recursively by setting n red0 = n 0 , then n red i = min( n i , n red i − 1 + 1) for i = 1 , . . .

Original: , L − 1 , and ﬁnally n red L = n L .
Revised: , L − 1 , and ﬁnally n red L = n L .

Original: We abbreviate Rep ( Q L , n ) by Rep ( n ) and Rep ( Q L , n red ) by Rep ( n red ) .
Revised: We abbreviate Rep ( Q L , n ) by Rep ( n ) and Rep ( Q L , n red ) by Rep ( n red ) .

Original: We also had in Section 5.2 the inclusion ι : Rep ( n red ) (cid:44) → Rep ( n ) , which can be understood in terms of padding matrices with rows and columns of zeros.
Revised: We also had in Section 4.2 the inclusion ι : Rep ( n red ) (cid:44) → Rep ( n ) , which can be understood in terms of padding matrices with rows and columns of zeros.

Original: Another description is as follows.
Revised: Another description is as follows.

Original: Let
Revised: Let

Original: X =
Revised: X =

Original: ( X 1 , . . .
Revised: ( X 1 , . . .

Original: , X L ) ∈
Revised: , X L ) ∈

Original: Rep ( n red ) .
Revised: Rep ( n red ) .

Original: Then ι ( X ) i = inc i ◦ X i ◦ (cid:101) π i − 1 .
Revised: Then ι ( X ) i = inc i ◦ X i ◦ (cid:101) π i − 1 .

Original: Indeed, post-composing X i with inc i amounts to adding n i − n red i rows of zeros on the bottom, while pre-composing X i with (cid:101)
Revised: Indeed, post-composing X i with inc i amounts to adding n i − n red i rows of zeros on the bottom, while pre-composing X i with (cid:101)

Original: π
Revised: π

Original: i −amounts to adding n i − 1 − n red
Revised: i −amounts to adding n i − 1 − n red

Original: i − 1 columns of zeros on the right.
Revised: i − 1 columns of zeros on the right.

Original: Fix a tuple h
Revised: Fix a tuple h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: , h L ) of functions
Revised: , h L ) of functions

Original: h i : R → R .
Revised: h i : R → R .

Original: As in Construction 5.2, we attach a radial neural network (and hence a neural function) to every representation of Q L .
Revised: As in Construction 4.2, we attach a radial neural network (and hence a neural function) to every representation of Q L .

Original: As a reminder, we restate Theorem 5.3.
Revised: As a reminder, we restate Theorem 4.3.

Original: Theorem E.5.
Revised: Theorem E.5.

Original: (Theorem 5.3) Let n be a dimension vector for Q L and ﬁx h as above.
Revised: (Theorem 4.3) Let n be a dimension vector for Q L and ﬁx h as above.

Original: For any
Revised: For any

Original: W ∈ Rep ( n ) there exist:
Revised: W ∈ Rep ( n ) there exist:

Original: [Equation]
Revised: [Equation]

Original: , R L ) ∈ Rep ( n red ) , and
Revised: , R L ) ∈ Rep ( n red ) , and

Original: U ∈ Rep ( n ) , such that:
Revised: U ∈ Rep ( n ) , such that:

Original: The matrices R 1 , . . .
Revised: The matrices R 1 , . . .

Original: , R L − 1 are upper triangular.
Revised: , R L − 1 are upper triangular.

Original: The following equality holds: W = Q · ι ( R ) + U 3.
Revised: The following equality holds: W = Q · ι ( R ) + U 3.

Original: The neural networks deﬁned by W and R have identical neural functions: f ( W , h )
Revised: The neural networks deﬁned by W and R have identical neural functions: f ( W , h )

Original: =
Revised: =

Original: f ( R , h ) .
Revised: f ( R , h ) .

Original: Before stating the proof, we ﬁrst adopt all notational conventions of Section 5.2, especially those listed after the statement of Algorithm 1.
Revised: Before stating the proof, we ﬁrst adopt all notational conventions of Section 4.2, especially those listed after the statement of Algorithm 1.

Original: Additionally, we note the projection maps:
Revised: Additionally, we note the projection maps:

Original: [Equation]
Revised: [Equation]

Original: onto the ﬁrst n red i and 1 + n red i coordinates, respectively.
Revised: onto the ﬁrst n red i and 1 + n red i coordinates, respectively.

Original: We also have the identity:
Revised: We also have the identity:

Original: [Equation]
Revised: [Equation]

Original: Next, we collect the following basic identities related to the map ext n : R n → R n +1 (see Deﬁnition
Revised: Next, we collect the following basic identities related to the map ext n : R n → R n +1 (see Deﬁnition

Original: B.1) taking ( v 1 , . .
Revised: B.1) taking ( v 1 , . .

Original: . , v n ) to (1 , v 1 , . . .
Revised: . , v n ) to (1 , v 1 , . . .

Original: , v n ) .
Revised: , v n ) .

Original: Their proofs are elementary.
Revised: Their proofs are elementary.

Original: Lemma E.6.
Revised: Lemma E.6.

Original: For m ≤ n , we have: ext n ◦ inc m,n = inc m +1 ,n +1 ◦ ext m .
Revised: For m ≤ n , we have: ext n ◦ inc m,n = inc m +1 ,n +1 ◦ ext m .

Original: Lemma E.7.
Revised: Lemma E.7.

Original: For any n × m matrix A , we have 7 : ext n ◦ A
Revised: For any n × m matrix A , we have 7 : ext n ◦ A

Original: = (cid:20) 

Revised: = (cid:20)

Original: A (cid:21) ◦ ext m .
Revised: A (cid:21) ◦ ext m .

Original: Proof of Theorem 5.3.
Revised: Proof of Theorem 4.3.

Original: We claim that the outputs Q , R , and U of Algorithm 1 satisfy the required conditions.
Revised: We claim that the outputs Q , R , and U of Algorithm 1 satisfy the required conditions.

Original: The matrices R 1 , . . .
Revised: The matrices R 1 , . . .

Original: , R L − 1 are upper-triangular by construction, so the ﬁrst part of the theorem holds.
Revised: , R L − 1 are upper-triangular by construction, so the ﬁrst part of the theorem holds.

Original: To show that the neural functions of ( W , h ) and ( R , h ) coincide, we ﬁrst show that, for all i = 1 , . .
Revised: To show that the neural functions of ( W , h ) and ( R , h ) coincide, we ﬁrst show that, for all i = 1 , . .

Original: , L , we have:
Revised: , L , we have:

Original: [Equation]
Revised: [Equation]

Original: where Q 0 = id n 0 and Q L = id n L .
Revised: where Q 0 = id n 0 and Q L = id n L .

Original: The justiﬁcation is a computation:
Revised: The justiﬁcation is a computation:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst equality uses the deﬁnition of (cid:102) W i in terms of W i ; the second uses the commutativity properties of ext stated in Lemmas E.6 and E.7; the third uses the deﬁnitions of Q i , R i , and V i ; and the fourth uses the commutativity properties of radial functions as well as the deﬁnition of (cid:101) R i .
Revised: The ﬁrst equality uses the deﬁnition of (cid:102) W i in terms of W i ; the second uses the commutativity properties of ext stated in Lemmas E.6 and E.7; the third uses the deﬁnitions of Q i , R i , and V i ; and the fourth uses the commutativity properties of radial functions as well as the deﬁnition of (cid:101) R i .

Original: The fact that f ( W , h ) =
Revised: The fact that f ( W , h ) =

Original: f ( R , h ) now follows from the deﬁnition of the neural function and iterative applications of Equation E.2, noting that Q 0 ◦ inc 0 = id n 0 and Q L ◦ inc L = id n L .
Revised: f ( R , h ) now follows from the deﬁnition of the neural function and iterative applications of Equation E.2, noting that Q 0 ◦ inc 0 = id n 0 and Q L ◦ inc L = id n L .

Original: Finally, to show that W = Q · ι ( R ) + U , we compute, for i = 1 , . . .
Revised: Finally, to show that W = Q · ι ( R ) + U , we compute, for i = 1 , . . .

Original: , L :
Revised: , L :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: inc
Revised: inc

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst two equalities follow from deﬁnitions; the third uses the deﬁnitions of Q i , R i , and V i ; the fourth uses the distributive law; and the ﬁfth appeals to the Equation E.1.
Revised: The ﬁrst two equalities follow from deﬁnitions; the third uses the deﬁnitions of Q i , R i , and V i ; the fourth uses the distributive law; and the ﬁfth appeals to the Equation E.1.

Original: F P ROJECTED GRADIENT DESCENT
Revised: F P ROJECTED GRADIENT DESCENT

Original: In this section, we give a proof of Theorem 5.7, which relates projected gradient descent for a representation with dimension n to (usual) gradient descent for the corresponding reduced representation with dimension vector n red .
Revised: In this section, we give a proof of Theorem 4.7, which relates projected gradient descent for a representation with dimension n to (usual) gradient descent for the corresponding reduced representation with dimension vector n red .

Original: We adopt all notational conventions from Section 5.
Revised: We adopt all notational conventions from Section 4.

Original: In particular, we ﬁx:
Revised: In particular, we ﬁx:

Original: • a tuple h
Revised: • a tuple h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: , h L ) of functions, where h i : R → R for i = 1 , . . .
Revised: , h L ) of functions, where h i : R → R for i = 1 , . . .

Original: , L
Revised: , L

Original: These
Revised: These

Original: can be used to deﬁne a neural function f ( W , h ) (resp.
Revised: can be used to deﬁne a neural function f ( W , h ) (resp.

Original: f
Revised: f

Original: ( X , h ) )
Revised: ( X , h ) )

Original: for every representation W in Rep ( n ) (resp.
Revised: for every representation W in Rep ( n ) (resp.

Original: X in Rep ( n red ) ).
Revised: X in Rep ( n red ) ).

Original: See Construction 5.2 for more details.
Revised: See Construction 4.2 for more details.

Original: • a batch of training data { ( x j, y j)
Revised: • a batch of training data { ( x j y j

Original: [Equation]
Revised: [Equation]

Original: R n
Revised: R n

Original: [Equation]
Revised: [Equation]

Original: R n L
Revised: R n L

Original: [Equation]
Revised: [Equation]

Original: R n red × R n red L
Revised: R n red0 × R n red L

Original: [Equation]
Revised: [Equation]

Original: • a cost function C : R n L × R n L → R
Revised: • a cost function C : R n L × R n L → R

Original: As a result, we have loss functions on Rep ( n ) and Rep ( n red ) given by:
Revised: As a result, we have loss functions on Rep ( n ) and Rep ( n red ) given by:

Original: [Equation]
Revised: [Equation]

Original: as in Section 4.2.
Revised: as in Section 3.4.

Original: Fixing a learning rate η > 0 , there are resulting gradient descent maps on Rep ( n ) and Rep ( n red ) given by:
Revised: Fixing a learning rate η > 0 , there are resulting gradient descent maps on Rep ( n ) and Rep ( n red ) given by:

Original: [Equation]
Revised: [Equation]

Original: F. T HE INTERPOLATING SPACE
Revised: F. T HE INTERPOLATING SPACE

Original: In this section, we introduce a subspace Rep int ( n ) of Rep ( n ) , that, as we will later see, interpolates between Rep ( n red ) and Rep ( n ) .
Revised: In this section, we introduce a subspace Rep int ( n ) of Rep ( n ) , that, as we will later see, interpolates between Rep ( n red ) and Rep ( n ) .

Original: Recall the standard inclusions and projections, for i = 0 , 1 , . . .
Revised: Recall the standard inclusions and projections, for i = 0 , 1 , . . .

Original: , L : inc
Revised: , L : inc

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition F.1.
Revised: Deﬁnition F.1.

Original: Let Rep int ( n ) denote the subspace of Rep ( n ) consisting of those T =
Revised: Let Rep int ( n ) denote the subspace of Rep ( n ) consisting of those T =

Original: ( T 1 , . . .
Revised: ( T 1 , . . .

Original: , T L ) ∈
Revised: , T L ) ∈

Original: Rep ( n ) such that:
Revised: Rep ( n ) such that:

Original: [Equation]
Revised: [Equation]

Original: for i = 1 , . . .
Revised: for i = 1 , . . .

Original: , L .
Revised: , L .

Original: Write ι 1 : Rep int ( n ) (cid:44) →
Revised: Write ι 1 : Rep int ( n ) (cid:44) →

Original: Rep ( n ) for the natural inclusion.
Revised: Rep ( n ) for the natural inclusion.

Original: In other words, the space Rep int ( n ) consists of representations T =
Revised: In other words, the space Rep int ( n ) consists of representations T =

Original: ( T 1 , . . .
Revised: ( T 1 , . . .

Original: , T L ) ∈
Revised: , T L ) ∈

Original: Rep ( n ) for which the bottom left ( n i − n red i ) ×
Revised: Rep ( n ) for which the bottom left ( n i − n red i ) ×

Original: (1 + n red i − 1 ) block of T i is zero for each i :
Revised: (1 + n red i − 1 ) block of T i is zero for each i :

Original: T i =
Revised: T i =

Original: (cid:20) ∗ ∗ 0 ∗ (cid:21) .
Revised: (cid:20) ∗ ∗ 0 ∗ (cid:21) .

Original: We omit the proof of the following proposition, as it involves an elementary analysis of the workings of Algorithm 1.
Revised: We omit the proof of the following proposition, as it involves an elementary analysis of the workings of Algorithm 1.

Original: Proposition F.2.
Revised: Proposition F.2.

Original: Let W ∈ Rep ( n ) and let Q , R , and U be the outputs of Algorithm 1, so
Revised: Let W ∈ Rep ( n ) and let Q , R , and U be the outputs of Algorithm 1, so

Original: W = Q · R + U .
Revised: W = Q · R + U .

Original: Then the representation Q − 1 · W = ι ( R ) + Q − 1 · U belongs to Rep int ( n ) .
Revised: Then the representation Q − 1 · W = ι ( R ) + Q − 1 · U belongs to Rep int ( n ) .

Original: Deﬁnition F.3.
Revised: Deﬁnition F.3.

Original: Deﬁne a map
Revised: Deﬁne a map

Original: [Equation]
Revised: [Equation]

Original: by taking T ∈ Rep ( n ) to the representation whose i -th slot is
Revised: by taking T ∈ Rep ( n ) to the representation whose i -th slot is

Original: [Equation]
Revised: [Equation]

Original: It is straightforward to check that q 1 is well-deﬁned, i.e., that the resulting representation belongs to Rep int ( n ) and that q 1 is a surjective linear map.
Revised: It is straightforward to check that q 1 is well-deﬁned, i.e., that the resulting representation belongs to Rep int ( n ) and that q 1 is a surjective linear map.

Original: The transpose of q 1 is the inclusion ι 1 .
Revised: The transpose of q 1 is the inclusion ι 1 .

Original: We summarize the situation in the following diagram:
Revised: We summarize the situation in the following diagram:

Original: [Equation]
Revised: [Equation]

Original: We observe that the composition q 1 ◦ ι is the identity on Rep int ( n ) , while the endomorphism ι 1 ◦ qof Rep ( n ) takes a representation W ∈ Rep ( n ) and, for each i , zeros all entries in the bottom left ( n i − n red i ) ×
Revised: We observe that the composition q 1 ◦ ι is the identity on Rep int ( n ) , while the endomorphism ι 1 ◦ qof Rep ( n ) takes a representation W ∈ Rep ( n ) and, for each i , zeros all entries in the bottom left ( n i − n red i ) ×

Original: (1 + n red i − 1 ) submatrix of W i .
Revised: (1 + n red i − 1 ) submatrix of W i .

Original: F. P ROJECTED GRADIENT DESCENT AND THE QR DECOMPOSITION
Revised: F. P ROJECTED GRADIENT DESCENT AND THE QR DECOMPOSITION

Original: In this section, we deﬁne the projected gradient descent map and state a theorem relating projected gradient descent on Rep ( n ) with usual gradient descent on Rep ( n red ) .
Revised: In this section, we deﬁne the projected gradient descent map and state a theorem relating projected gradient descent on Rep ( n ) with usual gradient descent on Rep ( n red ) .

Original: Deﬁnition F.4.
Revised: Deﬁnition F.4.

Original: The projected gradient descent map on Rep ( n ) with respect to Rep int ( n ) at learning rate η < 0 is deﬁned as:
Revised: The projected gradient descent map on Rep ( n ) with respect to Rep int ( n ) at learning rate η < 0 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where Pr = ι 1 ◦ q 1 .
Revised: where Pr = ι 1 ◦ q 1 .

Original: One veriﬁes easily that the above deﬁnition of projected gradient descent is the same as that given in
Revised: One veriﬁes easily that the above deﬁnition of projected gradient descent is the same as that given in

Original: Deﬁnition 5.6.
Revised: Deﬁnition 4.6.

Original: To reiterate, while all entries of each matrix W i in the representation W contribute to the computation of the gradient ∇ W L , only those not in the bottom left submatrix get updated under the projected gradient descent map γ proj .
Revised: To reiterate, while all entries of each matrix W i in the representation W contribute to the computation of the gradient ∇ W L , only those not in the bottom left submatrix get updated under the projected gradient descent map γ proj .

Original: We now restate Theorem 5.7.
Revised: We now restate Theorem 4.7.

Original: To ease notation, we identify elements of Rep ( n red ) with their images in Rep ( n ) under ι .
Revised: To ease notation, we identify elements of Rep ( n red ) with their images in Rep ( n ) under ι .

Original: Theorem F.5.
Revised: Theorem F.5.

Original: (Theorem 5.7)
Revised: (Theorem 4.7)

Original: Let W ∈ Rep ( n ) and let Q , R , and U be the outputs of Algorithm 1.
Revised: Let W ∈ Rep ( n ) and let Q , R , and U be the outputs of Algorithm 1.

Original: Fix a learning rate η > 0 .
Revised: Fix a learning rate η > 0 .

Original: For any k ≥ 0 , we have:
Revised: For any k ≥ 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We summarize this result in the following diagram.
Revised: We summarize this result in the following diagram.

Original: The left horizontal maps indicate the addition of Q − 1 · U , the right horizontal arrows indicate the action of Q , and the vertical maps are various versions of gradient descent.
Revised: The left horizontal maps indicate the addition of Q − 1 · U , the right horizontal arrows indicate the action of Q , and the vertical maps are various versions of gradient descent.

Original: The shaded regions indicate the (smallest) vector space to which the various representations naturally belong.
Revised: The shaded regions indicate the (smallest) vector space to which the various representations naturally belong.

Original: [Equation]
Revised: [Equation]

Original: Remark F.6.
Revised: Remark F.6.

Original: The map Rep ( n ) → Rep ( n ) taking W to Q · γ proj ( Q − 1 · W ) is a projected gradient descent map on Rep ( n ) with respect to the subspace of Rep ( n ) formed by translating Rep int ( n ) by the action of Q , i.e. the subspace Q · Rep int ( n ) =
Revised: The map Rep ( n ) → Rep ( n ) taking W to Q · γ proj ( Q − 1 · W ) is a projected gradient descent map on Rep ( n ) with respect to the subspace of Rep ( n ) formed by translating Rep int ( n ) by the action of Q , i.e. the subspace Q · Rep int ( n ) =

Original: { Q · T | T ∈ Rep int ( n ) } .
Revised: { Q · T | T ∈ Rep int ( n ) } .

Original: F. P ROOF OF T HEOREM 5. 
Revised: F. P ROOF OF T HEOREM 4.

Original: We begin by explaining the sense in which Rep int ( n ) interpolates between Rep ( n ) and Rep ( n red ) .
Revised: We begin by explaining the sense in which Rep int ( n ) interpolates between Rep ( n ) and Rep ( n red ) .

Original: One extends Diagram F.1 as follows:
Revised: One extends Diagram F.1 as follows:

Original: [Equation]
Revised: [Equation]

Original: • The map ι 2 : Rep ( n red ) (cid:44) → Rep int ( n ) takes X to the representation whose i -th coordinate is inc i ◦
Revised: • The map ι 2 : Rep ( n red ) (cid:44) → Rep int ( n ) takes X to the representation whose i -th coordinate is inc i ◦

Original: X i ◦ (cid:101)
Revised: X i ◦ (cid:101)

Original: π i − 1 .
Revised: π i − 1 .

Original: It is straightforward to check that ι 2 is a well-deﬁned injective linear map.
Revised: It is straightforward to check that ι 2 is a well-deﬁned injective linear map.

Original: • The map q 2 : Rep int ( n ) (cid:16) Rep ( n red ) takes T to the representation whose i -th slot is π i ◦ T i ◦ (cid:102) inc i − 1 .
Revised: • The map q 2 : Rep int ( n ) (cid:16) Rep ( n red ) takes T to the representation whose i -th slot is π i ◦ T i ◦ (cid:102) inc i − 1 .

Original: It is straightforward to check that q 2 is a surjective linear map.
Revised: It is straightforward to check that q 2 is a surjective linear map.

Original: The transpose of q 2 is the inclusion ι 2 .
Revised: The transpose of q 2 is the inclusion ι 2 .

Original: Lemma F.7.
Revised: Lemma F.7.

Original: We have the following:
Revised: We have the following:

Original: The inclusion ι : Rep ( n red ) (cid:44) → Rep ( n ) coincides with the composition ι 1 ◦ ι 2 , and commutes with the loss functions: Rep ( n red ) (cid:31) (cid:127) ι 1 ◦ ι 2 = ι (cid:47) (cid:47) L red (cid:35) (cid:35)
Revised: The inclusion ι : Rep ( n red ) (cid:44) → Rep ( n ) coincides with the composition ι 1 ◦ ι 2 , and commutes with the loss functions: Rep ( n red ) (cid:31) (cid:127) ι 1 ◦ ι 2 = ι (cid:47) (cid:47) L red (cid:35) (cid:35)

Original: Rep ( n ) L (cid:124)
Revised: Rep ( n ) L (cid:124)

Original: (cid:124) R 2.
Revised: (cid:124) R 2.

Original: The following diagram commutes: Rep int ( n ) q(cid:47) (cid:47) (cid:47) (cid:47) (cid:95) (cid:127) ι 1 (cid:15) (cid:15) Rep ( n red )
Revised: The following diagram commutes: Rep int ( n ) q(cid:47) (cid:47) (cid:47) (cid:47) (cid:95) (cid:127) ι 1 (cid:15) (cid:15) Rep ( n red )

Original: L red (cid:15) (cid:15) Rep ( n ) L (cid:47) (cid:47) R 3.
Revised: L red (cid:15) (cid:15) Rep ( n ) L (cid:47) (cid:47) R 3.

Original: For any T ∈ Rep int ( n ) , we have: q 1 (cid:0) ∇ ι 1 ( T ) L (cid:1) =
Revised: For any T ∈ Rep int ( n ) , we have: q 1 (cid:0) ∇ ι 1 ( T ) L (cid:1) =

Original: ι 2 (cid:0) ∇ q 2 ( T ) L red (cid:1) .
Revised: ι 2 (cid:0) ∇ q 2 ( T ) L red (cid:1) .

Original: Proof.
Revised: Proof.

Original: The identity ι = ι 1 ◦ ι 2 follows directly from deﬁnitions.
Revised: The identity ι = ι 1 ◦ ι 2 follows directly from deﬁnitions.

Original: To prove the commutativity of the diagram, it is enough to show that, for any X in Rep ( n red ) , the neural functions of X and ι ( X ) coincide.
Revised: To prove the commutativity of the diagram, it is enough to show that, for any X in Rep ( n red ) , the neural functions of X and ι ( X ) coincide.

Original: This follows easily from the fact that, for i = 1 , . . .
Revised: This follows easily from the fact that, for i = 1 , . . .

Original: , L , we have:
Revised: , L , we have:

Original: [Equation]
Revised: [Equation]

Original: For the second claim, let T ∈ Rep int ( n ) .
Revised: For the second claim, let T ∈ Rep int ( n ) .

Original: It sufﬁces to show that ι 1 ( T ) and q 2 ( T ) have the same neural function.
Revised: It sufﬁces to show that ι 1 ( T ) and q 2 ( T ) have the same neural function.

Original: The key computation is:
Revised: The key computation is:

Original: [Equation]
Revised: [Equation]

Original: which uses the fact that ( id n i − inc i ◦ π i )
Revised: which uses the fact that ( id n i − inc i ◦ π i )

Original: ◦ T i ◦ (cid:102) inc i − 1 = 0 , or, equivalently, inc i ◦ π
Revised: ◦ T i ◦ (cid:102) inc i − 1 = 0 , or, equivalently, inc i ◦ π

Original: i ◦ T i ◦ (cid:102) inc i − 1 =
Revised: i ◦ T i ◦ (cid:102) inc i − 1 =

Original: T i ◦ (cid:102) inc i − 1 .
Revised: T i ◦ (cid:102) inc i − 1 .

Original: Applying this relation successively starting with the second-to-last layer ( i =
Revised: Applying this relation successively starting with the second-to-last layer ( i =

Original: L − 1) and ending in the ﬁrst ( i = 1) , one obtains the result.
Revised: L − 1) and ending in the ﬁrst ( i = 1) , one obtains the result.

Original: For the last claim, one computes ∇ T ( L ◦ ι 1 ) in two different ways.
Revised: For the last claim, one computes ∇ T ( L ◦ ι 1 ) in two different ways.

Original: The ﬁrst way is:
Revised: The ﬁrst way is:

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that ι 1 is a linear map whose transpose is q 1 .
Revised: where we use the fact that ι 1 is a linear map whose transpose is q 1 .

Original: The second way uses the commutative diagram of the second part of the Lemma:
Revised: The second way uses the commutative diagram of the second part of the Lemma:

Original: [Equation]
Revised: [Equation]

Original: We also use the fact that q 2 is a linear map whose transpose is ι 2 .
Revised: We also use the fact that q 2 is a linear map whose transpose is ι 2 .

Original: Proof of Theorem 5.7.
Revised: Proof of Theorem 4.7.

Original: The action of Q ∈ O ( n ) on Rep ( n ) is an orthogonal transformation, so the ﬁrst claim follows from Proposition C.2.
Revised: The action of Q ∈ O ( n ) on Rep ( n ) is an orthogonal transformation, so the ﬁrst claim follows from Proposition C.2.

Original: For the second claim, it sufﬁces to consider the case η = 1 .
Revised: For the second claim, it sufﬁces to consider the case η = 1 .

Original: The general case follows similarly.
Revised: The general case follows similarly.

Original: We proceed by induction.
Revised: We proceed by induction.

Original: The base case k = 0 amounts to
Revised: The base case k = 0 amounts to

Original: Theorem 5.3.
Revised: Theorem 4.3.

Original: For the induction step, we set
Revised: For the induction step, we set

Original: [Equation]
Revised: [Equation]

Original: Each Z ( k ) belongs to Rep int ( n ) , so i 1 ( Z ( k ) )
Revised: Each Z ( k ) belongs to Rep int ( n ) , so i 1 ( Z ( k ) )

Original: =
Revised: =

Original: Z ( k ) .
Revised: Z ( k ) .

Original: Moreover, q 2 (cid:0) Z ( k ) (cid:1) =
Revised: Moreover, q 2 (cid:0) Z ( k ) (cid:1) =

Original: γ k red ( R ) .
Revised: γ k red ( R ) .

Original: We compute:
Revised: We compute:

Original: [Equation]
Revised: [Equation]

Original: where the second equality uses the induction hypothesis, the third invokes the deﬁnition of γ proj , the fourth uses the relation between the gradient and orthogonal transformations, the ﬁfth and sixth use
Revised: where the second equality uses the induction hypothesis, the third invokes the deﬁnition of γ proj , the fourth uses the relation between the gradient and orthogonal transformations, the ﬁfth and sixth use

Original: Lemma F.7 above, and the last uses the deﬁnition of γ red .
Revised: Lemma F.7 above, and the last uses the deﬁnition of γ red .

Original: F. E XAMPLE
Revised: F. E XAMPLE

Original: We now discuss an example where projected gradient descent does not match usual gradient descent.
Revised: We now discuss an example where projected gradient descent does not match usual gradient descent.

Original: Let n = (1 , 3 , 1) be a dimension vector for the neural quiver Q L .
Revised: Let n = (1 , 3 , 1) be a dimension vector for the neural quiver Q L .

Original: The space of representations with this dimension vector is 10-dimensional:
Revised: The space of representations with this dimension vector is 10-dimensional:

Original: [Equation]
Revised: [Equation]

Original: We identify a representation
Revised: We identify a representation

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with the point p
Revised: with the point p

Original: = ( a, b, c, d, e, f, g, h, i, j ) in R 10 .
Revised: = ( a, b, c, d, e, f, g, h, i, j ) in R 10 .

Original: The action of the orthogonal group O ( n ) =
Revised: The action of the orthogonal group O ( n ) =

Original: O (3) on Rep ( Q L , n ) (cid:39) R 10 can be expressed as:
Revised: O (3) on Rep ( Q L , n ) (cid:39) R 10 can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Consider the function 8 :
Revised: Consider the function 8 :

Original: [Equation]
Revised: [Equation]

Original: By the product rule, we have:
Revised: By the product rule, we have:

Original: [Equation]
Revised: [Equation]

Original: One easily checks that L ( Q · p ) =
Revised: One easily checks that L ( Q · p ) =

Original: L ( p ) and that ∇ Q · p L = Q · ∇ p L for any Q ∈ O (3) .
Revised: L ( p ) and that ∇ Q · p L = Q · ∇ p L for any Q ∈ O (3) .

Original: The interpolating space is the subspace of Rep ( Q L , n ) (cid:39) R 8 with e = f = 0 .
Revised: The interpolating space is the subspace of Rep ( Q L , n ) (cid:39) R 8 with e = f = 0 .

Original: Suppose p (cid:48)
Revised: Suppose p (cid:48)

Original: = ( a, b, c, d, 0 , 0 , g, h, i, j ) belongs to the interpolating space.
Revised: = ( a, b, c, d, 0 , 0 , g, h, i, j ) belongs to the interpolating space.

Original: Then the gradient is ∇ p
Revised: Then the gradient is ∇ p

Original: [Equation]
Revised: [Equation]

Original: L =
Revised: L =

Original: ( h, h, i, i, j, j, 1 , a + b, c
Revised: ( h, h, i, i, j, j, 1 , a + b, c

Original: + d, 0) which does not belong to the interpolating space.
Revised: + d, 0) which does not belong to the interpolating space.

Original: So one step of usual gradient descent, with learning rate η > 0 yields:
Revised: So one step of usual gradient descent, with learning rate η > 0 yields:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, one step of projected gradient descent yields:
Revised: On the other hand, one step of projected gradient descent yields:

Original: [Equation]
Revised: [Equation]

Original: Direct computation shows that the difference between the evaluation of L after one step of gradient descent and the evaluation of L after one step of projected gradient descent is:
Revised: Direct computation shows that the difference between the evaluation of L after one step of gradient descent and the evaluation of L after one step of projected gradient descent is:

Original: [Equation]
Revised: [Equation]

Original: G D ISCUSSION
Revised: G D ISCUSSION

Original: In this section, we ﬁrst consider how our framework specializes to the case of no bias, and then how it generalizes to shifts within the radial functions.
Revised: In this section, we ﬁrst consider how our framework specializes to the case of no bias, and then how it generalizes to shifts within the radial functions.

Original: G. S PECIAL CASE : N O - BIAS VERSION
Revised: G. S PECIAL CASE : N O - BIAS VERSION

Original: We now consider neural networks with only linear maps between successive layers, rather than the more general setting of afﬁne maps.
Revised: We now consider neural networks with only linear maps between successive layers, rather than the more general setting of afﬁne maps.

Original: In other words, there are no bias vectors.
Revised: In other words, there are no bias vectors.

Original: Let L be a positive integer.
Revised: Let L be a positive integer.

Original: The no-bias neural quiver Q L is the following quiver with L
Revised: The no-bias neural quiver Q L is the following quiver with L

Original: +vertices:
Revised: +vertices:

Original: [Equation]
Revised: [Equation]

Original: A representation of this quiver with dimension vector n = ( n 0 , . .
Revised: A representation of this quiver with dimension vector n = ( n 0 , . .

Original: . , n L ) consists of a linear map from R n i − 1 to R n i for i = 1 , . . .
Revised: . , n L ) consists of a linear map from R n i − 1 to R n i for i = 1 , . . .

Original: , L ; hence Rep ( Q L , n ) =
Revised: , L ; hence Rep ( Q L , n ) =

Original: (cid:76) Li
Revised: (cid:76) Li

Original: =
Revised: =

Original: Hom( R n i − 1 , R n i ) .
Revised: Hom( R n i − 1 , R n i ) .

Original: The corresponding no-bias reduced dimension vector n red = ( n red 0 , n red 1 , . . . , n red L ) is deﬁned recursively by setting n red 0 = n 0 , then n red
Revised: The corresponding no-bias reduced dimension vector n red = ( n red 0 , n red 1 , . . . , n red L ) is deﬁned recursively by setting n red 0 = n 0 , then n red

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: − 1 ) for i = 1 , . . .
Revised: − 1 ) for i = 1 , . . .

Original: , L − 1 , and ﬁnally n red L = n L .
Revised: , L − 1 , and ﬁnally n red L = n L .

Original: Since n red i ≤ n i for all i , we have an obvious inclusion Rep ( Q L , n red ) (cid:44) → Rep ( Q L , n ) and identify
Revised: Since n red i ≤ n i for all i , we have an obvious inclusion Rep ( Q L , n red ) (cid:44) → Rep ( Q L , n ) and identify

Original: Rep ( Q L , n red ) with its image in Rep ( Q L , n ) .
Revised: Rep ( Q L , n red ) with its image in Rep ( Q L , n ) .

Original: Given functions h
Revised: Given functions h

Original: = ( h 1 , . . .
Revised: = ( h 1 , . . .

Original: , h L ) , one adapts Construction 5.2 to deﬁne a radial neural network for every representation of Q L , where the trainable parameters deﬁne linear maps rather than afﬁne maps.
Revised: , h L ) , one adapts Construction 4.2 to deﬁne a radial neural network for every representation of Q L , where the trainable parameters deﬁne linear maps rather than afﬁne maps.

Original: Proposition G.1.
Revised: Proposition G.1.

Original: Theorem 5.3 holds with the neural quiver replaced by the no-bias neural quiver
Revised: Theorem 4.3 holds with the neural quiver replaced by the no-bias neural quiver

Original: Q L and the reduced dimension vector replaced by the no-bias reduced dimension vector n red .
Revised: Q L and the reduced dimension vector replaced by the no-bias reduced dimension vector n red .

Original: We illustrate an example of the reduction for no-bias radial neural networks in Figure 4 (which is the same as Figure 2).
Revised: We illustrate an example of the reduction for no-bias radial neural networks in Figure 5 (which is the same as Figure 3).

Original: Versions of Algorithm 1 and Theorem 5.7 also hold in the no-bias case, where one uses projected gradient descent with respect to the subspace Rep int ( Q L , n ) of representations T having the lower left ( n i − n red i ) ×
Revised: Versions of Algorithm 1 and Theorem 4.7 also hold in the no-bias case, where one uses projected gradient descent with respect to the subspace Rep int ( Q L , n ) of representations T having the lower left ( n i − n red i ) ×

Original: ( n red i − 1 ) block of each T i equal to zero.
Revised: ( n red i − 1 ) block of each T i equal to zero.

Original: R
Revised: R

Original: R 4 R R 4 R
Revised: R 4 R R 4 R

Original: R  1 . 20 .
Revised: R  1 . 20 .

Original: 5 − .
Revised: 5 − .

Original: 1  σ  0 .
Revised: 1  σ  0 .

Original: 7 − .
Revised: 7 − .

Original: 4 − .
Revised: 4 − .

Original: 2 − .
Revised: 2 − .

Original: 5 − .
Revised: 5 − .

Original: 4  σ (cid:0) .
Revised: 4  σ (cid:0) .

Original: 7 . 4 (cid:1)
Revised: 7 . 4 (cid:1)

Original: R
Revised: R

Original: R R
Revised: R R

Original: R 4 R
Revised: R 4 R

Original: R (cid:0) − 1 . 31 (cid:1) σ  − 0 .
Revised: R (cid:0) − 1 . 31 (cid:1) σ  − 0 .

Original: 2 − 0 .
Revised: 2 − 0 .

Original: 3 − 0 .
Revised: 3 − 0 .

Original: 1 − 1 .
Revised: 1 − 1 .

Original: 1  σ (cid:0) .
Revised: 1  σ (cid:0) .

Original: 7 . 4 (cid:1)
Revised: 7 . 4 (cid:1)

Original: R
Revised: R

Original: R R R R
Revised: R R R R

Original: R (cid:0) − 1 . 31 (cid:1) σ (cid:0) 1 . 17 (cid:1) σ (cid:0) − 0 .
Revised: R (cid:0) − 1 . 31 (cid:1) σ (cid:0) 1 . 17 (cid:1) σ (cid:0) − 0 .

Original: 5 (cid:1)
Revised: 5 (cid:1)

Original: Example G.2.
Revised: Example G.2.

Original: We give an example where projected gradient descent does not match usual gradient descent.
Revised: We give an example where projected gradient descent does not match usual gradient descent.

Original: This example is a simpler version of that appearing in F.4.
Revised: This example is a simpler version of that appearing in F.4.

Original: Let n = (1 , 2 , 1) be a dimension vector for the no-bias quiver Q L .
Revised: Let n = (1 , 2 , 1) be a dimension vector for the no-bias quiver Q L .

Original: The space of representations with this dimension vector is 4-dimensional:
Revised: The space of representations with this dimension vector is 4-dimensional:

Original: [Equation]
Revised: [Equation]

Original: We identify a representation
Revised: We identify a representation

Original: [Equation]
Revised: [Equation]

Original: with the point p = ( a, b, c, d ) in R 4 .
Revised: with the point p = ( a, b, c, d ) in R 4 .

Original: The action of the orthogonal group O ( n ) =
Revised: The action of the orthogonal group O ( n ) =

Original: O (2) on
Revised: O (2) on

Original: Rep ( Q L , n ) (cid:39) R 4 is the diagonal action:
Revised: Rep ( Q L , n ) (cid:39) R 4 is the diagonal action:

Original: [Equation]
Revised: [Equation]

Original: To be explicit: if Q =
Revised: To be explicit: if Q =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: sin( θ ) cos( θ ) (cid:21) is an orthogonal transformation, then Q · ( a, b, c, d ) =
Revised: sin( θ ) cos( θ ) (cid:21) is an orthogonal transformation, then Q · ( a, b, c, d ) =

Original: ( a cos( θ ) − b sin( θ ) , a sin( θ ) + b cos( θ ) , c cos( θ ) − d sin( θ ) , c sin( θ ) +
Revised: ( a cos( θ ) − b sin( θ ) , a sin( θ ) + b cos( θ ) , c cos( θ ) − d sin( θ ) , c sin( θ ) +

Original: d cos( θ )) .
Revised: d cos( θ )) .

Original: Consider the function 9 :
Revised: Consider the function 9 :

Original: [Equation]
Revised: [Equation]

Original: By the product rule, we have:
Revised: By the product rule, we have:

Original: [Equation]
Revised: [Equation]

Original: One easily checks that L ( Q · p ) =
Revised: One easily checks that L ( Q · p ) =

Original: L ( p ) and that ∇ Q · p L = Q · ∇ p L for any Q ∈ O (2) .
Revised: L ( p ) and that ∇ Q · p L = Q · ∇ p L for any Q ∈ O (2) .

Original: The reduced dimension vector is n red =
Revised: The reduced dimension vector is n red =

Original: so Rep
Revised: so Rep

Original: [Equation]
Revised: [Equation]

Original: R.
Revised: R.

Original: The reduced version of L is given by:
Revised: The reduced version of L is given by:

Original: [Equation]
Revised: [Equation]

Original: The QR decomposition of the representation W corresponding to p =
Revised: The QR decomposition of the representation W corresponding to p =

Original: ( a, b, c, d ) is:
Revised: ( a, b, c, d ) is:

Original: [Equation]
Revised: [Equation]

Original: a, b, c, d ) = Q ·
Revised: a, b, c, d ) = Q ·

Original: [Equation]
Revised: [Equation]

Original: (cid:19) where Q = 1 √ a 2 + b(cid:20) a − b b a (cid:21) .
Revised: (cid:19) where Q = 1 √ a 2 + b(cid:20) a − b b a (cid:21) .

Original: The interpolating space is the subspace of Rep ( Q L , n ) (cid:39) R 4 with b = 0 .
Revised: The interpolating space is the subspace of Rep ( Q L , n ) (cid:39) R 4 with b = 0 .

Original: Suppose p (cid:48)
Revised: Suppose p (cid:48)

Original: = ( a, 0 , c, d ) belongs to the interpolating space.
Revised: = ( a, 0 , c, d ) belongs to the interpolating space.

Original: Then the gradient is
Revised: Then the gradient is

Original: [Equation]
Revised: [Equation]

Original: which does not belong to the interpolating space.
Revised: which does not belong to the interpolating space.

Original: So one step of usual gradient descent, with learning rate η > 0 yields:
Revised: So one step of usual gradient descent, with learning rate η > 0 yields:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, one step of projected gradient descent yields:
Revised: On the other hand, one step of projected gradient descent yields:

Original: [Equation]
Revised: [Equation]

Original: The difference between the evaluation of L after one step of gradient descent and the evaluation of
Revised: The difference between the evaluation of L after one step of gradient descent and the evaluation of

Original: L after one step of projected gradient descent is:
Revised: L after one step of projected gradient descent is:

Original: [Equation]
Revised: [Equation]

Original: G. G ENERALIZATION : R ADIAL NEURAL NETWORKS WITH SHIFTS
Revised: G. G ENERALIZATION : R ADIAL NEURAL NETWORKS WITH SHIFTS

Original: In this section we consider radial neural networks with an extra trainable parameter in each layer which shifts the radial function.
Revised: In this section we consider radial neural networks with an extra trainable parameter in each layer which shifts the radial function.

Original: Adding such parameters allows for more ﬂexibility in the model, and (as shown in Theorem G.4) the QR decomposition of Theorem 5.3 holds for such radial neural networks.
Revised: Adding such parameters allows for more ﬂexibility in the model, and (as shown in Theorem G.4) the QR decomposition of Theorem 4.3 holds for such radial neural networks.

Original: Let h : R → R be a function 10 .
Revised: Let h : R → R be a function 10 .

Original: For any n ≥ 1 and any t ∈ R , the corresponding shifted radial function on R n is given by:
Revised: For any n ≥ 1 and any t ∈ R , the corresponding shifted radial function on R n is given by:

Original: [Equation]
Revised: [Equation]

Original: The following deﬁnition is a modiﬁcation of Deﬁnition
Revised: The following deﬁnition is a modiﬁcation of Deﬁnition

Original: Deﬁnition G.3.
Revised: Deﬁnition G.3.

Original: A radial neural network with shifts consists of the following data:
Revised: A radial neural network with shifts consists of the following data:

Original: Hyperparameters.
Revised: Hyperparameters.

Original: A positive integer L and a dimension vector n = ( n 0 , n 1 , n 2 , . .
Revised: A positive integer L and a dimension vector n = ( n 0 , n 1 , n 2 , . .

Original: . , n L ) for the neural quiver Q L . 2.
Revised: . , n L ) for the neural quiver Q L . 2.

Original: Trainable parameters.
Revised: Trainable parameters.

Original: (a) A representation W =
Revised: (a) A representation W =

Original: ( W 1 , . . .
Revised: ( W 1 , . . .

Original: , W L ) of the quiver Q L with dimension vector n .
Revised: , W L ) of the quiver Q L with dimension vector n .

Original: So, for i = 1 , . . .
Revised: So, for i = 1 , . . .

Original: , L , we have a matrix W
Revised: , L , we have a matrix W

Original: i ∈
Revised: i ∈

Original: Hom( R 1+ n i − 1 , R n i ) .
Revised: Hom( R 1+ n i − 1 , R n i ) .

Original: (b) A vector of shifts t
Revised: (b) A vector of shifts t

Original: = ( t 1 , t 2 , . . .
Revised: = ( t 1 , t 2 , . . .

Original: , t L ) ∈ R L .
Revised: , t L ) ∈ R L .

Original: Radial activation functions.
Revised: Radial activation functions.

Original: A tuple h =
Revised: A tuple h =

Original: ( h 1 , h 2 , . . .
Revised: ( h 1 , h 2 , . . .

Original: , h L ) , where h i : R → R .
Revised: , h L ) , where h i : R → R .

Original: The activation function in the i -th layer is given by a i
Revised: The activation function in the i -th layer is given by a i

Original: = h ( n i ,t i ) : R n i
Revised: = h ( n i ,t i ) : R n i

Original: → R n i for i = 1 , . . .
Revised: → R n i for i = 1 , . . .

Original: , L .
Revised: , L .

Original: The neural function of a radial neural network with shifts is deﬁned as:
Revised: The neural function of a radial neural network with shifts is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:102) W i
Revised: where (cid:102) W i

Original: =
Revised: =

Original: W i ◦ ext n i − 1 : R n i − 1 → R n i is the afﬁne map corresponding to W i .
Revised: W i ◦ ext n i − 1 : R n i − 1 → R n i is the afﬁne map corresponding to W i .

Original: The trainable parameters form the vector space Rep ( n ) ⊕ R L , and the loss function of a batch of training data is deﬁned as
Revised: The trainable parameters form the vector space Rep ( n ) ⊕ R L , and the loss function of a batch of training data is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We have the gradient descent map:
Revised: We have the gradient descent map:

Original: [Equation]
Revised: [Equation]

Original: which updates the entries of both W and t .
Revised: which updates the entries of both W and t .

Original: The group O ( n ) =
Revised: The group O ( n ) =

Original: O ( n 1 ) × · · · × O ( n L − 1 ) acts on
Revised: O ( n 1 ) × · · · × O ( n L − 1 ) acts on

Original: Rep ( n ) as usual (see Section 5.1), and on R L trivially.
Revised: Rep ( n ) as usual (see Section 4.1), and on R L trivially.

Original: The neural function is unchanged by this action.
Revised: The neural function is unchanged by this action.

Original: We conclude that the O ( n ) action on Rep ( n ) ⊕ R L commutes with gradient descent γ .
Revised: We conclude that the O ( n ) action on Rep ( n ) ⊕ R L commutes with gradient descent γ .

Original: We now state a generalization of Theorem 5.3 for the case of radial neural networks with shifts.
Revised: We now state a generalization of Theorem 4.3 for the case of radial neural networks with shifts.

Original: We omit a proof, as it uses the same techniques as the proof of Theorem 5.3.
Revised: We omit a proof, as it uses the same techniques as the proof of Theorem 4.3.

Original: Theorem G.4.
Revised: Theorem G.4.

Original: Let n be a dimension vector for Q L and
Revised: Let n be a dimension vector for Q L and

Original: ﬁx functions h
Revised: ﬁx functions h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: , h L ) as above.
Revised: , h L ) as above.

Original: For any W ∈ Rep ( n ) there exist:
Revised: For any W ∈ Rep ( n ) there exist:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: such that:
Revised: such that:

Original: The matrices R 1 , . . .
Revised: The matrices R 1 , . . .

Original: , R L − 1 are upper triangular.
Revised: , R L − 1 are upper triangular.

Original: The following equality holds: ( W , t ) =
Revised: The following equality holds: ( W , t ) =

Original: Q · ( R , t )
Revised: Q · ( R , t )

Original: + ( U , 0 ) .
Revised: + ( U , 0 ) .

Original: The neural functions deﬁned by ( W , t , h ) and ( R , t , h ) coincide: f ( W , t , h ) =
Revised: The neural functions deﬁned by ( W , t , h ) and ( R , t , h ) coincide: f ( W , t , h ) =

Original: f ( R , t , h ) .
Revised: f ( R , t , h ) .

Original: One can use the output of Algorithm 1 to obtain the Q , R , and U appearing in Theorem G.4.
Revised: One can use the output of Algorithm 1 to obtain the Q , R , and U appearing in Theorem G.4.

Original: Theorem 5.7 also generalizes to the setting of radial neural networks with shifts, using projected gradient descent with respect to the subspace Rep int ( n ) ⊕ R L of Rep ( n ) ⊕ R L .
Revised: Theorem 4.7 also generalizes to the setting of radial neural networks with shifts, using projected gradient descent with respect to the subspace Rep int ( n ) ⊕ R L of Rep ( n ) ⊕ R L .

Original: H C ATEGORICAL FORMULATION
Revised: H C ATEGORICAL FORMULATION

Original: In this appendix, we summarize a category-theoretic approach toward the main results of the paper.
Revised: In this appendix, we summarize a category-theoretic approach toward the main results of the paper.

Original: While there is no substantial difference in the proofs, the language of category theory provides conceptual clarity that leads to generalizations of these results.
Revised: While there is no substantial difference in the proofs, the language of category theory provides conceptual clarity that leads to generalizations of these results.

Original: References for category theory include Pierce (1991); Dummit & Foote (2003).
Revised: References for category theory include Pierce (1991); Dummit & Foote (2003).

Original: H. T HE CATEGORY OF QUIVER REPRESENTATIONS
Revised: H. T HE CATEGORY OF QUIVER REPRESENTATIONS

Original: In this section, we recall the category of representations of a quiver.
Revised: In this section, we recall the category of representations of a quiver.

Original: Background references include
Revised: Background references include

Original: Kirillov Jr (2016); Nakajima et al. (1998).
Revised: Kirillov Jr (2016); Nakajima et al. (1998).

Original: As in Section 4, let Q
Revised: As in Section 3, let Q

Original: =
Revised: =

Original: ( I, E ) be a quiver with source and target maps s, t : E → I , and let d : I → Z ≥ 0 ,
Revised: ( I, E ) be a quiver with source and target maps s, t : E → I , and let d : I → Z ≥ 0 ,

Original: i
Revised: i

Original: (cid:55)→ d i be a dimension vector for Q .
Revised: (cid:55)→ d i be a dimension vector for Q .

Original: Recall that a representation of Q with dimension vector d consists of a tuple A =
Revised: Recall that a representation of Q with dimension vector d consists of a tuple A =

Original: ( A e ) e ∈ E of linear maps, where
Revised: ( A e ) e ∈ E of linear maps, where

Original: [Equation]
Revised: [Equation]

Original: Let A and B be representations of the quiver Q , with dimension vectors d = dim( A ) and k = dim( B ) .
Revised: Let A and B be representations of the quiver Q , with dimension vectors d = dim( A ) and k = dim( B ) .

Original: A morphism of representations from A to B consists of the data of a linear map
Revised: A morphism of representations from A to B consists of the data of a linear map

Original: [Equation]
Revised: [Equation]

Original: for every i ∈ I , subject to the condition that the following diagram commutes for every e ∈ E :
Revised: for every i ∈ I , subject to the condition that the following diagram commutes for every e ∈ E :

Original: [Equation]
Revised: [Equation]

Original: The resulting category R ( Q ) is known as the category of representations of Q .
Revised: The resulting category R ( Q ) is known as the category of representations of Q .

Original: H. T HE CATEGORY I ( Q L )
Revised: H. T HE CATEGORY I ( Q L )

Original: In this section, we deﬁne a certain subcategory I ( Q L ) of the category R ( Q L ) .
Revised: In this section, we deﬁne a certain subcategory I ( Q L ) of the category R ( Q L ) .

Original: Its objects are the same as the objects of R ( Q L ) , that is, representations of the neural quiver, while morphisms in
Revised: Its objects are the same as the objects of R ( Q L ) , that is, representations of the neural quiver, while morphisms in

Original: I ( Q L ) are given by isometries.
Revised: I ( Q L ) are given by isometries.

Original: Let L be a positive integer, and recall the neural quiver Q L from Section 4.1:
Revised: Let L be a positive integer, and recall the neural quiver Q L from Section 3.2:

Original: [Equation]
Revised: [Equation]

Original: (cid:47) (cid:47) • • (cid:55) (cid:55) (cid:61) (cid:61) (cid:97) (cid:97) (cid:103) (cid:103)
Revised: [Table]

Original: As a reminder, the vertices in the top row are indexed from i = 0 to i =
Revised: As a reminder, the vertices in the top row are indexed from i = 0 to i =

Original: L , and we only consider dimension vectors whose value at the bias vertex is equal to 1 .
Revised: L , and we only consider dimension vectors whose value at the bias vertex is equal to 1 .

Original: So a dimension vector for Q L will refer to a tuple n = ( n 0 , n 1 , . .
Revised: So a dimension vector for Q L will refer to a tuple n = ( n 0 , n 1 , . .

Original: . , n L ) .
Revised: . , n L ) .

Original: Recall the isomorphism
Revised: Recall the isomorphism

Original: Rep ( Q L , n ) (cid:39)
Revised: Rep ( Q L , n ) (cid:39)

Original: L (cid:77) i =
Revised: L (cid:77) i =

Original: Hom( R 1+ n i − 1 , R n i ) from Lemma 4.1.
Revised: Hom( R 1+ n i − 1 , R n i ) from Lemma 3.1.

Original: We denote a representation of Q L as a tuple W =
Revised: We denote a representation of Q L as a tuple W =

Original: ( W i )
Revised: ( W i )

Original: Li
Revised: Li

Original: =1 , where each W i belongs to Hom( R 1+ n i − 1 , R n i ) .
Revised: =1 , where each W i belongs to Hom( R 1+ n i − 1 , R n i ) .

Original: Let X =
Revised: Let X =

Original: ( X i )
Revised: ( X i )

Original: Li
Revised: Li

Original: =1 be a representation of Q L with dimension vector m .
Revised: =1 be a representation of Q L with dimension vector m .

Original: Tracing through the proof of Lemma 4.1 and the deﬁnitions in Section H.1, we see that a morphism α : X → W in R ( Q L ) consists of the following data:
Revised: Tracing through the proof of Lemma 3.1 and the deﬁnitions in Section H.1, we see that a morphism α : X → W in R ( Q L ) consists of the following data:

Original: • a linear map α
Revised: • a linear map α

Original: i : R m i → R n i , for i = 0 , 1 , . . .
Revised: i : R m i → R n i , for i = 0 , 1 , . . .

Original: , L , and • a scalar α b ∈ R at the bias vertex,
Revised: , L , and • a scalar α b ∈ R at the bias vertex,

Original: making the following diagram commute, for i = 1 , . .
Revised: making the following diagram commute, for i = 1 , . .

Original: , L :
Revised: , L :

Original: R 1+ m i −
Revised: R 1+ m i −

Original: X i (cid:47) (cid:47) (cid:20) α b 0α i −(cid:21) (cid:15) (cid:15)
Revised: X i (cid:47) (cid:47) (cid:20) α b 0α i −(cid:21) (cid:15) (cid:15)

Original: R m i α i (cid:15) (cid:15)
Revised: R m i α i (cid:15) (cid:15)

Original: R 1+ n i −
Revised: R 1+ n i −

Original: W i (cid:47) (cid:47) R n i
Revised: W i (cid:47) (cid:47) R n i

Original: Deﬁnition H.1.
Revised: Deﬁnition H.1.

Original: We deﬁne a subcategory I ( Q L ) of R ( Q L ) as follows.
Revised: We deﬁne a subcategory I ( Q L ) of R ( Q L ) as follows.

Original: The objects of I ( Q L ) are the same as the objects of R ( Q L ) , that is, representations of Q L .
Revised: The objects of I ( Q L ) are the same as the objects of R ( Q L ) , that is, representations of Q L .

Original: Let X and W be such representations, with dimension vectors m and n , respectively.
Revised: Let X and W be such representations, with dimension vectors m and n , respectively.

Original: A morphism α : X → W in R ( Q L ) belongs to
Revised: A morphism α : X → W in R ( Q L ) belongs to

Original: I ( Q L ) if the following hold:
Revised: I ( Q L ) if the following hold:

Original: • n 0 = m 0 and α 0 is the identity on R n 0 , • n L = m L and α L is the identity on R n L , • for i = 1 , . . .
Revised: • n 0 = m 0 and α 0 is the identity on R n 0 , • n L = m L and α L is the identity on R n L , • for i = 1 , . . .

Original: , L − 1 , the linear map α
Revised: , L − 1 , the linear map α

Original: i : R m i → R n i is norm-preserving, i.e. |
Revised: i : R m i → R n i is norm-preserving, i.e. |

Original: α i ( v ) | = | v | for all v ∈ R
Revised: α i ( v ) | = | v | for all v ∈ R

Original: m i , and • α b = 1 .
Revised: m i , and • α b = 1 .

Original: Remark H.2.
Revised: Remark H.2.

Original: A norm-preserving map is called an isometry , which explains the notation I ( Q L ) .
Revised: A norm-preserving map is called an isometry , which explains the notation I ( Q L ) .

Original: Lemma H.3.
Revised: Lemma H.3.

Original: We have:
Revised: We have:

Original: The category I
Revised: The category I

Original: ( Q L ) is a well-deﬁned subcategory of R L . 2.
Revised: ( Q L ) is a well-deﬁned subcategory of R L . 2.

Original: Let α : X → W be a morphism in I ( Q L ) .
Revised: Let α : X → W be a morphism in I ( Q L ) .

Original: For i = 0 , 1 , . . .
Revised: For i = 0 , 1 , . . .

Original: , L , the linear map α
Revised: , L , the linear map α

Original: i : R m i → R n i is injective.
Revised: i : R m i → R n i is injective.

Original: Proof.
Revised: Proof.

Original: The claims follow from the facts that (1) the composition of two norm-preserving maps is norm-preserving, and (2) any linear norm-preserving map is injective.
Revised: The claims follow from the facts that (1) the composition of two norm-preserving maps is norm-preserving, and (2) any linear norm-preserving map is injective.

Original: Deﬁnition H.4.
Revised: Deﬁnition H.4.

Original: Let α : X → W be a morphism in I ( Q L ) , and let m = dim( X ) and n = dim( W ) be the dimension vectors.
Revised: Let α : X → W be a morphism in I ( Q L ) , and let m = dim( X ) and n = dim( W ) be the dimension vectors.

Original: An orthogonal factorization of α i is an element Q =
Revised: An orthogonal factorization of α i is an element Q =

Original: ( Q 1 , . . .
Revised: ( Q 1 , . . .

Original: , Q L − 1 ) of O ( n ) =
Revised: , Q L − 1 ) of O ( n ) =

Original: O ( n 1 ) × · · · × O ( n L − 1 ) such that
Revised: O ( n 1 ) × · · · × O ( n L − 1 ) such that

Original: [Equation]
Revised: [Equation]

Original: for i = 1 , . . .
Revised: for i = 1 , . . .

Original: , L − 1 .
Revised: , L − 1 .

Original: The correction term corresponding to an orthogonal factorization is:
Revised: The correction term corresponding to an orthogonal factorization is:

Original: [Equation]
Revised: [Equation]

Original: Q · X .
Revised: Q · X .

Original: The correction term belongs to Rep ( Q L , n ) .
Revised: The correction term belongs to Rep ( Q L , n ) .

Original: Remark H.5.
Revised: Remark H.5.

Original: Orthogonal factorizations always exist, since any norm-preserving linear map R m
Revised: Orthogonal factorizations always exist, since any norm-preserving linear map R m

Original: → R n can be written as the composition Q ◦ inc m,n for some orthogonal Q ∈ O ( n ) .
Revised: → R n can be written as the composition Q ◦ inc m,n for some orthogonal Q ∈ O ( n ) .

Original: Remark H.6.
Revised: Remark H.6.

Original: Suppose α is a morphism in I ( Q L ) .
Revised: Suppose α is a morphism in I ( Q L ) .

Original: For i ∈ { 1 , . . .
Revised: For i ∈ { 1 , . . .

Original: , L − 1 } , the map α
Revised: , L − 1 } , the map α

Original: i is an isomorphism if and only if m i = n i .
Revised: i is an isomorphism if and only if m i = n i .

Original: In this case, the choice of Q i is unique and U
Revised: In this case, the choice of Q i is unique and U

Original: i +

Revised: i +

Original: = 0 .
Revised: = 0 .

Original: Conversely, α i is not an isomorphism if and only if m i < n i .
Revised: Conversely, α i is not an isomorphism if and only if m i < n i .

Original: In this case, there are O ( n i − m i ) choices for Q i .
Revised: In this case, there are O ( n i − m i ) choices for Q i .

Original: Fix functions h
Revised: Fix functions h

Original: i : R → R for i = 1 , . . .
Revised: i : R → R for i = 1 , . . .

Original: , L .
Revised: , L .

Original: Hence we obtain radial functions h ( n ) i : R n → R n for any n ≥ 1 .
Revised: Hence we obtain radial functions h ( n ) i : R n → R n for any n ≥ 1 .

Original: We group the h i into a tuple h
Revised: We group the h i into a tuple h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: . , h L ) .
Revised: . , h L ) .

Original: Given h , we attach a neural network (and hence a neural function) to every object in I ( Q L ) as in Construction 5.2.
Revised: Given h , we attach a neural network (and hence a neural function) to every object in I ( Q L ) as in Construction 4.2.

Original: Proposition H.7.
Revised: Proposition H.7.

Original: Let X and W be a representations of Q L .
Revised: Let X and W be a representations of Q L .

Original: Suppose there is a morphism in I ( Q L ) from X to W .
Revised: Suppose there is a morphism in I ( Q L ) from X to W .

Original: Then the neural functions of the radial neural networks ( X , h ) and ( W , h ) coincide: f
Revised: Then the neural functions of the radial neural networks ( X , h ) and ( W , h ) coincide: f

Original: ( X , h ) =
Revised: ( X , h ) =

Original: f ( W , h ) .
Revised: f ( W , h ) .

Original: Sketch of proof.
Revised: Sketch of proof.

Original: The key is to show that, for i = 1 , . . .
Revised: The key is to show that, for i = 1 , . . .

Original: , L , we have:
Revised: , L , we have:

Original: [Equation]
Revised: [Equation]

Original: where (cid:102) W i and (cid:101)
Revised: where (cid:102) W i and (cid:101)

Original: X i are the afﬁne map corresponding to W i and X i .
Revised: X i are the afﬁne map corresponding to W i and X i .

Original: The ﬁrst step in the veriﬁcation of this identity is to choose an orthogonal factorization of α .
Revised: The ﬁrst step in the veriﬁcation of this identity is to choose an orthogonal factorization of α .

Original: The rest of the proof proceeds along the same lines as the proof of Equation E.2 in Section E.2.
Revised: The rest of the proof proceeds along the same lines as the proof of Equation E.2 in Section E.2.

Original: H. P ROJECTED GRADIENT DESCENT : SET - UP
Revised: H. P ROJECTED GRADIENT DESCENT : SET - UP

Original: In this section, we collect notation necessary to state results about projected gradient descent.
Revised: In this section, we collect notation necessary to state results about projected gradient descent.

Original: Let m =
Revised: Let m =

Original: ( m 0 , . . .
Revised: ( m 0 , . . .

Original: , m L ) and n = ( n 0 , . .
Revised: , m L ) and n = ( n 0 , . .

Original: . , n L ) be dimension vectors for Q L .
Revised: . , n L ) be dimension vectors for Q L .

Original: We write m (cid:22) n if:
Revised: We write m (cid:22) n if:

Original: • m 0 = n 0 and m L = n L , • m i ≤ n i for i = 1 , . . .
Revised: • m 0 = n 0 and m L = n L , • m i ≤ n i for i = 1 , . . .

Original: , L − 1 .
Revised: , L − 1 .

Original: Consequently, if α : X → W is a morphism in I ( Q L ) , then the dimension vectors satisfy dim( X ) (cid:22) dim( W ) .
Revised: Consequently, if α : X → W is a morphism in I ( Q L ) , then the dimension vectors satisfy dim( X ) (cid:22) dim( W ) .

Original: For m (cid:22) n , we make the following abbreviations, for i = 0 , 1 , . . .
Revised: For m (cid:22) n , we make the following abbreviations, for i = 0 , 1 , . . .

Original: , L :
Revised: , L :

Original: [Equation]
Revised: [Equation]

Original: Using these maps, one deﬁnes an inclusion ι : Rep ( m ) (cid:44) → Rep ( n ) taking X =
Revised: Using these maps, one deﬁnes an inclusion ι : Rep ( m ) (cid:44) → Rep ( n ) taking X =

Original: ( X 1 , . . .
Revised: ( X 1 , . . .

Original: , X L ) ∈
Revised: , X L ) ∈

Original: Rep ( m ) to the representation with ι ( X ) i = inc i ◦
Revised: Rep ( m ) to the representation with ι ( X ) i = inc i ◦

Original: X i ◦ (cid:101)
Revised: X i ◦ (cid:101)

Original: π i − 1 .
Revised: π i − 1 .

Original: Recall the functions
Revised: Recall the functions

Original: h
Revised: h

Original: = ( h 1 , . .
Revised: = ( h 1 , . .

Original: . , h L ) .
Revised: . , h L ) .

Original: As in Construction 5.2, these deﬁne activation functions (cid:16) h ( n 1 ), . .
Revised: As in Construction 4.2, these deﬁne activation functions (cid:16) h ( n 1 ), . .

Original: , h ( n L ) L (cid:17) (resp.
Revised: , h ( n L ) L (cid:17) (resp.

Original: (cid:16) h ( m 1 ), . .
Revised: (cid:16) h ( m 1 ), . .

Original: , h ( m L ) L (cid:17) ) for a representation with dimension n (resp. m ).
Revised: , h ( m L ) L (cid:17) ) for a representation with dimension n (resp. m ).

Original: Finally, we ﬁx a batch of training data X = { ( x j , y j ) } ⊆ R n 0 × R n L =
Revised: Finally, we ﬁx a batch of training data X = { ( x j , y j ) } ⊆ R n 0 × R n L =

Original: R m 

Revised: R m

Original: × R m L .
Revised: × R m L .

Original: Using the activation functions deﬁned by h , we have loss functions on Rep ( n ) and Rep ( m ) (see Section 4.2):
Revised: Using the activation functions deﬁned by h , we have loss functions on Rep ( n ) and Rep ( m ) (see Section 3.4):

Original: [Equation]
Revised: [Equation]

Original: Fixing a learning rate η > 0 , there are resulting gradient descent maps on Rep ( n ) and Rep ( m ) given by:
Revised: Fixing a learning rate η > 0 , there are resulting gradient descent maps on Rep ( n ) and Rep ( m ) given by:

Original: [Equation]
Revised: [Equation]

Original: The veriﬁcation of the following lemma is analogous to the proof of Part 1 of Lemma F.7.
Revised: The veriﬁcation of the following lemma is analogous to the proof of Part 1 of Lemma F.7.

Original: Lemma H.8.
Revised: Lemma H.8.

Original: We have that L n ◦ ι = L m .
Revised: We have that L n ◦ ι = L m .

Original: H. T HE INTERPOLATING SPACE
Revised: H. T HE INTERPOLATING SPACE

Original: We ﬁrst deﬁne a space that interpolates between Rep ( m ) and Rep ( n ) .
Revised: We ﬁrst deﬁne a space that interpolates between Rep ( m ) and Rep ( n ) .

Original: The discussion of this section is completely analogous to that in Sections F.1 and F.2.
Revised: The discussion of this section is completely analogous to that in Sections F.1 and F.2.

Original: Deﬁnition H.9.
Revised: Deﬁnition H.9.

Original: Let Rep int ( m , n ) denote the subspace of Rep ( n ) consisting of those T
Revised: Let Rep int ( m , n ) denote the subspace of Rep ( n ) consisting of those T

Original: =
Revised: =

Original: ( T 1 , . . .
Revised: ( T 1 , . . .

Original: , T L ) ∈
Revised: , T L ) ∈

Original: Rep ( n ) such that, for i = 1 , . . .
Revised: Rep ( n ) such that, for i = 1 , . . .

Original: , L , we have:
Revised: , L , we have:

Original: [Equation]
Revised: [Equation]

Original: Just as in Section F.1, the space Rep int ( m , n ) consists of representations T
Revised: Just as in Section F.1, the space Rep int ( m , n ) consists of representations T

Original: = ( T 1 , . .
Revised: = ( T 1 , . .

Original: , T L ) ∈
Revised: , T L ) ∈

Original: Rep ( n ) for which the bottom left ( n i − m i ) × (1 + n red i − 1 ) block of T i is zero for each i .
Revised: Rep ( n ) for which the bottom left ( n i − m i ) × (1 + n red i − 1 ) block of T i is zero for each i .

Original: Consider the maps:
Revised: Consider the maps:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: • The map ι 1 : Rep int ( m , n ) (cid:44) → Rep ( n ) is the natural inclusion.
Revised: • The map ι 1 : Rep int ( m , n ) (cid:44) → Rep ( n ) is the natural inclusion.

Original: • The map q 1 : Rep ( n ) → Rep int ( m , n ) takes T ∈ Rep ( n ) to the representation whose i -th slot is
Revised: • The map q 1 : Rep ( n ) → Rep int ( m , n ) takes T ∈ Rep ( n ) to the representation whose i -th slot is

Original: [Equation]
Revised: [Equation]

Original: • The map ι 2 : Rep ( m ) (cid:44) → Rep int ( m , n ) is deﬁned by taking X to the representation whose i -th coordinate is inc i ◦
Revised: • The map ι 2 : Rep ( m ) (cid:44) → Rep int ( m , n ) is deﬁned by taking X to the representation whose i -th coordinate is inc i ◦

Original: X i ◦ (cid:101)
Revised: X i ◦ (cid:101)

Original: π i − 1 .
Revised: π i − 1 .

Original: • The map q 2 : Rep int ( n ) (cid:16) Rep ( m ) is deﬁned by taking T to the representation whose i -th slot is π i ◦ T i ◦ (cid:102) inc i − 1 .
Revised: • The map q 2 : Rep int ( n ) (cid:16) Rep ( m ) is deﬁned by taking T to the representation whose i -th slot is π i ◦ T i ◦ (cid:102) inc i − 1 .

Original: Deﬁnition H.10.
Revised: Deﬁnition H.10.

Original: The projected gradient descent map on Rep ( n ) with respect to Rep int ( m , n ) and learning rate η > 0 is deﬁned as:
Revised: The projected gradient descent map on Rep ( n ) with respect to Rep int ( m , n ) and learning rate η > 0 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: We now state the main result of this appendix.
Revised: We now state the main result of this appendix.

Original: We omit a proof, as it follows the same ideas as the proof of Theorem 5.7 given in Section F.3.
Revised: We omit a proof, as it follows the same ideas as the proof of Theorem 4.7 given in Section F.3.

Original: The main difference is that all appearances of n red must be replaced by m .
Revised: The main difference is that all appearances of n red must be replaced by m .