Original: D EEP T RANSFORMERS WITHOUT S HORTCUTS : M ODIFYING S ELF - ATTENTION FOR F AITHFUL S IGNAL P ROPAGATION
Revised: D EEP T RANSFORMERS WITHOUT S HORTCUTS : M ODIFYING S ELF - ATTENTION FOR F AITHFUL S IGNAL P ROPAGATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles are poorly understood.
Revised: Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles are poorly understood.

Original: Recent approaches such as Deep Kernel Shaping have made progress towards reducing our reliance on them, using insights from wide NN kernel theory to improve signal propagation in vanilla DNNs (which we define as networks without skips or normalisation layers).
Revised: Recent approaches such as Deep Kernel Shaping have made progress towards reducing our reliance on them, using insights from wide NN kernel theory to improve signal propagation in vanilla DNNs (which we define as networks without skips or normalisation layers).

Original: However, these approaches are incompatible with the self-attention layers present in transformers, whose kernels are intrinsically more complicated to analyse and control.
Revised: However, these approaches are incompatible with the self-attention layers present in transformers, whose kernels are intrinsically more complicated to analyse and control.

Original: And so the question remains: is it possible to train deep vanilla transformers?
Revised: And so the question remains: is it possible to train deep vanilla transformers?

Original: We answer this question in the affirmative by designing several approaches that use combinations of parameter initialisations, bias matrices and location-dependent rescaling to achieve faithful signal propagation in vanilla transformers.
Revised: We answer this question in the affirmative by designing several approaches that use combinations of parameter initialisations, bias matrices and location-dependent rescaling to achieve faithful signal propagation in vanilla transformers.

Original: Our methods address several intricacies specific to signal propagation in transformers, including the interaction with positional encoding and causal masking.
Revised: Our methods address several intricacies specific to signal propagation in transformers, including the interaction with positional encoding and causal masking.

Original: In experiments on WikiText-103 and C4, our approaches enable deep transformers without normalisation to train at speeds matching their standard counterparts, and deep vanilla transformers to reach the same performance as standard ones after abouttimes more iterations.
Revised: In experiments on WikiText-103 and C4, our approaches enable deep transformers without normalisation to train at speeds matching their standard counterparts, and deep vanilla transformers to reach the same performance as standard ones after abouttimes more iterations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite numerous impressive successes, the practice of training deep neural networks (DNNs) has progressed to a large extent independently of theoretical justification.
Revised: Despite numerous impressive successes, the practice of training deep neural networks (DNNs) has progressed to a large extent independently of theoretical justification.

Original: Most successful modern DNN architectures rely on particular arrangements of skip connections and normalisation layers, but a general principle for how to use these components in new architectures (assuming they are even applicable) remains unknown, and their roles in existing ones are still not completely understood.
Revised: Most successful modern DNN architectures rely on particular arrangements of skip connections and normalisation layers, but a general principle for how to use these components in new architectures (assuming they are even applicable) remains unknown, and their roles in existing ones are still not completely understood.

Original: The residual architecture , arguably the most popular and successful of these, was first developed in the context of convolutional networks (CNNs)
Revised: The residual architecture , arguably the most popular and successful of these, was first developed in the context of convolutional networks (CNNs)

Original: (He et al., 2016), and later in self-attention networks yielding the ubiquitous transformer architecture (Vaswani et al., 2017).
Revised: (He et al., 2016), and later in self-attention networks yielding the ubiquitous transformer architecture (Vaswani et al., 2017).

Original: One proposed explanation for the success of residual architectures is that they have superior signal propagation compared to vanilla DNNs (e.g. Balduzzi et al., 2017; Xiao et al., 2018;
Revised: One proposed explanation for the success of residual architectures is that they have superior signal propagation compared to vanilla DNNs (e.g. Balduzzi et al., 2017; Xiao et al., 2018;

Original: Hayou et al., 2019; De & Smith, 2020;
Revised: Hayou et al., 2019; De & Smith, 2020;

Original: Martens et al., 2021), where signal propagation refers to the transmission of geometric information through the layers of a DNN, as represented by a kernel function (Daniely et al., 2016; Poole et al., 2016; Schoenholz et al., 2017).
Revised: Martens et al., 2021), where signal propagation refers to the transmission of geometric information through the layers of a DNN, as represented by a kernel function (Daniely et al., 2016; Poole et al., 2016; Schoenholz et al., 2017).

Original: Recently, using signal propagation principles to train DNNs at high depths, without the skip connections and/or normalisation layers found in residual architectures, has become an area of interest in the community.
Revised: Recently, using signal propagation principles to train DNNs at high depths, without the skip connections and/or normalisation layers found in residual architectures, has become an area of interest in the community.

Original: The reasons are two-fold.
Revised: The reasons are two-fold.

Original: First, it would validate the signal propagation hypothesis for the effectiveness of residual architectures, thus clarifying our understanding of DNN trainability.
Revised: First, it would validate the signal propagation hypothesis for the effectiveness of residual architectures, thus clarifying our understanding of DNN trainability.

Original: And second, it could lead to general principles and techniques for achieving trainability in DNNs beyond the residual paradigm, with the potential for improved or more efficient architectures.
Revised: And second, it could lead to general principles and techniques for achieving trainability in DNNs beyond the residual paradigm, with the potential for improved or more efficient architectures.

Original: For CNNs, Xiao et al.
Revised: For CNNs, Xiao et al.

Original: (2018) showed that improved signal propagation from better initialisation enables very deep vanilla networks to be effectively trained, although at significantly reduced speeds compared to residual networks.
Revised: (2018) showed that improved signal propagation from better initialisation enables very deep vanilla networks to be effectively trained, although at significantly reduced speeds compared to residual networks.

Original: Martens et al.
Revised: Martens et al.

Original: (2021) later proposed Deep Kernel Shaping (DKS) which uses activation function transformations to control signal propagation, achieving training
Revised: (2021) later proposed Deep Kernel Shaping (DKS) which uses activation function transformations to control signal propagation, achieving training

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: r d v a n ill a
Revised: r d v a n ill a

Original: Layer
Revised: Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: [Figure]
Revised: [Figure]

Original: n i f
Revised: n i f

Original: o r
Revised: o r

Original: [Figure]
Revised: [Figure]

Original: P
Revised: P

Original: [Figure]
Revised: [Figure]

Original: x p o n e n t i a l S P
Revised: x p o n e n t i a l S P

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: speed parity between vanilla and residual networks on ImageNet assuming the use of strong 2ndorder optimizers like K-FAC (Martens & Grosse, 2015).
Revised: speed parity between vanilla and residual networks on ImageNet assuming the use of strong 2ndorder optimizers like K-FAC (Martens & Grosse, 2015).

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022) extended ideas from
Revised: (2022) extended ideas from

Original: DKS to a larger class of activation functions, achieving near parity in terms of generalisation as well.
Revised: DKS to a larger class of activation functions, achieving near parity in terms of generalisation as well.

Original: The key quantity that is analysed in signal propagation is the DNN’s initialisation-time kernel, or more precisely, the approximate kernel given by the infinite width limit (Neal, 2012; Matthews et al., 2018; Lee et al., 2018;
Revised: The key quantity that is analysed in signal propagation is the DNN’s initialisation-time kernel, or more precisely, the approximate kernel given by the infinite width limit (Neal, 2012; Matthews et al., 2018; Lee et al., 2018;

Original: Yang, 2019).
Revised: Yang, 2019).

Original: For MLPs, and for CNNs that use a Delta-initialisation (Balduzzi et al., 2017; Xiao et al., 2018), this kernel can be written as a simple recursion over layers that involves only 2D functions, facilitating a straightforward analysis.
Revised: For MLPs, and for CNNs that use a Delta-initialisation (Balduzzi et al., 2017; Xiao et al., 2018), this kernel can be written as a simple recursion over layers that involves only 2D functions, facilitating a straightforward analysis.

Original: Unfortunately, the evolution of the kernel across layers of a transformer is more complicated, and as a result, existing approaches like DKS are not applicable to transformers (or indeed any architecture that contains self-attention layers).
Revised: Unfortunately, the evolution of the kernel across layers of a transformer is more complicated, and as a result, existing approaches like DKS are not applicable to transformers (or indeed any architecture that contains self-attention layers).

Original: More concretely, if X l ∈ R T × d denotes a length- T sequence of activations at layer l of a transformer, then the kernel matrix Σ l = X
Revised: More concretely, if X l ∈ R T × d denotes a length- T sequence of activations at layer l of a transformer, then the kernel matrix Σ l = X

Original: l X ⊤ l /d ∈ R T × T for layer l (or more precisely its limit as d → ∞ ), can be written as a function of the kernel matrix Σ l − 1 of the previous layer (Hron et al., 2020).
Revised: l X ⊤ l /d ∈ R T × T for layer l (or more precisely its limit as d → ∞ ), can be written as a function of the kernel matrix Σ l − 1 of the previous layer (Hron et al., 2020).

Original: In the case of self-attention layers, the relationship of Σ l on Σ l −cannot be simplified or decomposed into lower dimensional functions, leading to a recursion that is intrinsically high dimensional and harder to analyse or control.
Revised: In the case of self-attention layers, the relationship of Σ l on Σ l −cannot be simplified or decomposed into lower dimensional functions, leading to a recursion that is intrinsically high dimensional and harder to analyse or control.

Original: Analogously to the case of MLPs, where signal propagation is judged by looking at the behavior of the (one-dimensional) kernel, signal propagation in transformers can be judged by looking at the evolution of these (high-dimensional) kernel matrices through the layers of the network.
Revised: Analogously to the case of MLPs, where signal propagation is judged by looking at the behavior of the (one-dimensional) kernel, signal propagation in transformers can be judged by looking at the evolution of these (high-dimensional) kernel matrices through the layers of the network.

Original: One situation we must avoid is where the diagonal entries rapidly grow or shrink with depth, which corresponds to uncontrolled activation norms and can lead to saturated losses or numerical issues.
Revised: One situation we must avoid is where the diagonal entries rapidly grow or shrink with depth, which corresponds to uncontrolled activation norms and can lead to saturated losses or numerical issues.

Original: A more subtle form of signal degradation can occur where Σ l converges to a rank-1 matrix, which is known as rank collapse (Dong et al., 2021).
Revised: A more subtle form of signal degradation can occur where Σ l converges to a rank-1 matrix, which is known as rank collapse (Dong et al., 2021).

Original: Dong et al.
Revised: Dong et al.

Original: (2021) showed that skip connections are essential to avoid the collapsed state: skipless transformers quickly converge to rank collapse at large depths, which we corroborate in Fig.
Revised: (2021) showed that skip connections are essential to avoid the collapsed state: skipless transformers quickly converge to rank collapse at large depths, which we corroborate in Fig.

Original: 1 (top).
Revised: 1 (top).

Original: Moreover, Noci et al.
Revised: Moreover, Noci et al.

Original: (2022) showed that rank collapse may lead to zero gradients for certain parameters in attention layers, hindering the trainablility of deep transformers.
Revised: (2022) showed that rank collapse may lead to zero gradients for certain parameters in attention layers, hindering the trainablility of deep transformers.

Original: Thus, avoiding rank collapse is necessary for deep transformers to be trainable, and the question of whether one can train deep skipless transformers remains open.
Revised: Thus, avoiding rank collapse is necessary for deep transformers to be trainable, and the question of whether one can train deep skipless transformers remains open.

Original: In this work we address this question, demonstrating for the first time that it is possible to successfully train deep transformers without skip connections or normalisation layers.
Revised: In this work we address this question, demonstrating for the first time that it is possible to successfully train deep transformers without skip connections or normalisation layers.

Original: To do so, we study the problem of signal propagation and rank collapse in deep skipless transformers, and derive three approaches to prevent it in Section 3.
Revised: To do so, we study the problem of signal propagation and rank collapse in deep skipless transformers, and derive three approaches to prevent it in Section 3.

Original: Our methods use combinations of: 1) parameter initialisations, 2) bias matrices and 3) location-dependent rescaling, and highlight several intricacies specific to signal propagation in transformers, including the interaction with positional encoding and causal masking.
Revised: Our methods use combinations of: 1) parameter initialisations, 2) bias matrices and 3) location-dependent rescaling, and highlight several intricacies specific to signal propagation in transformers, including the interaction with positional encoding and causal masking.

Original: In Section 4, we empirically demonstrate that our approaches result in trainable deep skipless transformers.
Revised: In Section 4, we empirically demonstrate that our approaches result in trainable deep skipless transformers.

Original: On WikiText-103 and C4 datasets we show that using our main approach, Exponential Signal Preserving Attention (E-SPA), it is possible to match the training loss of standard transformers with our skipless ones by training for around 5 times longer.
Revised: On WikiText-103 and C4 datasets we show that using our main approach, Exponential Signal Preserving Attention (E-SPA), it is possible to match the training loss of standard transformers with our skipless ones by training for around 5 times longer.

Original: Moreover, by combining this approach with skip connections, we show that transformers without normalisation layers are able to match the training speed of standard ones.
Revised: Moreover, by combining this approach with skip connections, we show that transformers without normalisation layers are able to match the training speed of standard ones.

Original:  P ROBLEM SETTING
Revised:  P ROBLEM SETTING

Original: Transformer models The input to a transformer consists of a sequence x =( x i )
Revised: Transformer models The input to a transformer consists of a sequence x =( x i )

Original: Ti =1 over T locations consisting of tokens from a vocabulary V : x i ∈ { 1 , . .
Revised: Ti =1 over T locations consisting of tokens from a vocabulary V : x i ∈ { 1 , . .

Original: , | V |} .
Revised: , | V |} .

Original: The model takes this sequence, and using a trainable embedding matrix E ∈ R | V |× d , creates a matrix of vector representations
Revised: The model takes this sequence, and using a trainable embedding matrix E ∈ R | V |× d , creates a matrix of vector representations

Original: X 0 ∈ R T ×
Revised: X 0 ∈ R T ×

Original: d by performing a direct look-up for each location
Revised: d by performing a direct look-up for each location

Original: : [ X 0 ] i = E x i ∈ R d .
Revised: : [ X 0 ] i = E x i ∈ R d .

Original: After this, the sequence is successively transformed via a series of L “transformer blocks”, with independently initialised parameters.
Revised: After this, the sequence is successively transformed via a series of L “transformer blocks”, with independently initialised parameters.

Original: We will denote by X l ∈ R T ×
Revised: We will denote by X l ∈ R T ×

Original: d the output sequence for block l , and will sometimes refer to the rows of X l as representation vectors .
Revised: d the output sequence for block l , and will sometimes refer to the rows of X l as representation vectors .

Original: Each transformer block consists of two component blocks that both employ a standard residual structure (He et al., 2016), computing the sum of a residual branch which performs the main computation, and a shortcut branch (aka a skip connection) which just copies the block’s inputs to its output.
Revised: Each transformer block consists of two component blocks that both employ a standard residual structure (He et al., 2016), computing the sum of a residual branch which performs the main computation, and a shortcut branch (aka a skip connection) which just copies the block’s inputs to its output.

Original: The attention block, which is the first of these component blocks, applies layer normalisation (LN)
Revised: The attention block, which is the first of these component blocks, applies layer normalisation (LN)

Original: (Ba et al., 2016) or RMSNorm (Zhang & Sennrich, 2019), followed by multi-head attention (MHA), on its residual branch.
Revised: (Ba et al., 2016) or RMSNorm (Zhang & Sennrich, 2019), followed by multi-head attention (MHA), on its residual branch.

Original: The MLP block, which comes next, applies LN followed by a standard (typically shallow) MLP on its residual branch.
Revised: The MLP block, which comes next, applies LN followed by a standard (typically shallow) MLP on its residual branch.

Original: The LN and MLP operations are applied to each sequence element independently (using shared parameters), so that information is only communicated between sequence elements via the MHA operation (which we will define later).
Revised: The LN and MLP operations are applied to each sequence element independently (using shared parameters), so that information is only communicated between sequence elements via the MHA operation (which we will define later).

Original: In summary we have
Revised: In summary we have

Original: [Equation]
Revised: [Equation]

Original: where the shortcut and residual weights α, β are typically both 1 .
Revised: where the shortcut and residual weights α, β are typically both 1 .

Original: In this work we will focus on skipless transformers with α = 0 and β = 1 , and on vanilla transformers, which are skipless transformers without normalisation layers.
Revised: In this work we will focus on skipless transformers with α = 0 and β = 1 , and on vanilla transformers, which are skipless transformers without normalisation layers.

Original: For simplicity we will also devote much of our analysis to attention-only models, which are transformers without MLP blocks (so that ˆ X l = X l ).
Revised: For simplicity we will also devote much of our analysis to attention-only models, which are transformers without MLP blocks (so that ˆ X l = X l ).

Original: Note that we restrict our analysis to decoder-only transformers in this work, as they have a simpler structure which is easier to analyse, and are widely used in practice.
Revised: Note that we restrict our analysis to decoder-only transformers in this work, as they have a simpler structure which is easier to analyse, and are widely used in practice.

Original: Also note that Eq.
Revised: Also note that Eq.

Original: (1) corresponds to the “Pre-LN”
Revised: (1) corresponds to the “Pre-LN”

Original: (Baevski & Auli, 2018; Child et al., 2019) rather than the original “Post-LN” transformer (Wang et al., 2019).
Revised: (Baevski & Auli, 2018; Child et al., 2019) rather than the original “Post-LN” transformer (Wang et al., 2019).

Original: In Post-LN transformers, the normalisation operation is applied at the output of each MLP and attention block instead of at the beginning of each residual branch.
Revised: In Post-LN transformers, the normalisation operation is applied at the output of each MLP and attention block instead of at the beginning of each residual branch.

Original: Self-attention Given an input sequence X ∈ R T × d , the self-attention mechanism computes
Revised: Self-attention Given an input sequence X ∈ R T × d , the self-attention mechanism computes

Original: Attn
Revised: Attn

Original: [Equation]
Revised: [Equation]

Original: softmax
Revised: softmax

Original: [Equation]
Revised: [Equation]

Original: where the softmax function is applied row-wise.
Revised: where the softmax function is applied row-wise.

Original: Q ( X ) =
Revised: Q ( X ) =

Original: XW Q , K ( X ) =
Revised: XW Q , K ( X ) =

Original: XW K and V ( X ) =
Revised: XW K and V ( X ) =

Original: XW V denote the queries , keys and values respectively, with trainable parameters W Q , W K ∈
Revised: XW V denote the queries , keys and values respectively, with trainable parameters W Q , W K ∈

Original: R d × d k and W V ∈ R d × d v .
Revised: R d × d k and W V ∈ R d × d v .

Original: In practice, the attention mechanism, Eq.
Revised: In practice, the attention mechanism, Eq.

Original: (2), is applied over h “heads” (with independent parameters), giving rise to so-called multi-head attention :
Revised: (2), is applied over h “heads” (with independent parameters), giving rise to so-called multi-head attention :

Original: [Equation]
Revised: [Equation]

Original: where W O ∈ R hd v × d are trainable parameters and usually d k = d v = dh .
Revised: where W O ∈ R hd v × d are trainable parameters and usually d k = d v = dh .

Original: In this case, we can define W V =
Revised: In this case, we can define W V =

Original: Concat ( W V 1 , . .
Revised: Concat ( W V 1 , . .

Original: , W Vh ) ∈ R
Revised: , W Vh ) ∈ R

Original: d × d where W Vn denotes the value parameters for head n .
Revised: d × d where W Vn denotes the value parameters for head n .

Original: We focus our investigation to models that perform next-token prediction : at location i − 1 the model outputs a prediction over the identity of the i th target token, but using only information from input tokens 1 through i − 1 .
Revised: We focus our investigation to models that perform next-token prediction : at location i − 1 the model outputs a prediction over the identity of the i th target token, but using only information from input tokens 1 through i − 1 .

Original: This corresponds to using a causal masked attention with mask M ∈ R T × T satisfying M i,j = 1 { i ≥ j } , where the attention matrix A in Eq.
Revised: This corresponds to using a causal masked attention with mask M ∈ R T × T satisfying M i,j = 1 { i ≥ j } , where the attention matrix A in Eq.

Original: (2) is replaced with
Revised: (2) is replaced with

Original: [Equation]
Revised: [Equation]

Original: where Γ is a large positive constant that zeros the attention coefficients corresponding to future tokens, making A a lower triangular matrix.
Revised: where Γ is a large positive constant that zeros the attention coefficients corresponding to future tokens, making A a lower triangular matrix.

Original: Signal propagation in transformers As discussed in Section 1, Dong et al.
Revised: Signal propagation in transformers As discussed in Section 1, Dong et al.

Original: (2021) showed that deep skipless transformers suffer from rank collapse , where the kernel matrix converges in depth to have rank 1, and Noci et al.
Revised: (2021) showed that deep skipless transformers suffer from rank collapse , where the kernel matrix converges in depth to have rank 1, and Noci et al.

Original: (2022) showed that rank collapse can prevent trainability.
Revised: (2022) showed that rank collapse can prevent trainability.

Original: Moreover, Noci et al.
Revised: Moreover, Noci et al.

Original: (2022) demonstrated that rank collapse in transformers can occur in the absence of normalisation layers even with skip connections, and that downscaling the residual branch by setting β = 1 √ L can alleviate this issue.
Revised: (2022) demonstrated that rank collapse in transformers can occur in the absence of normalisation layers even with skip connections, and that downscaling the residual branch by setting β = 1 √ L can alleviate this issue.

Original: This latter observation is in line with previous findings concerning the benefits of downscaling residual weights in ResNets (Hanin & Rolnick, 2018; Arpit et al., 2019;
Revised: This latter observation is in line with previous findings concerning the benefits of downscaling residual weights in ResNets (Hanin & Rolnick, 2018; Arpit et al., 2019;

Original: Hayou et al., 2021; Bachlechner et al., 2021) and transformers (Zhang et al., 2019; Xu et al., 2020; Huang et al., 2020; Touvron et al., 2021; Wang et al., 2022).
Revised: Hayou et al., 2021; Bachlechner et al., 2021) and transformers (Zhang et al., 2019; Xu et al., 2020; Huang et al., 2020; Touvron et al., 2021; Wang et al., 2022).

Original: Davis et al.
Revised: Davis et al.

Original: (2021) showed that concatenation acts similarly to a downweighted skip as an alternative way to connect skip and residual branches.
Revised: (2021) showed that concatenation acts similarly to a downweighted skip as an alternative way to connect skip and residual branches.

Original: De & Smith (2020) noted that the interaction of standard skip connections and normalisations can also effectively downweight the residual branch to give better signal propagation properties, but only if the normalisation layer is placed on the residual branch, like in Pre-LN transformers.
Revised: De & Smith (2020) noted that the interaction of standard skip connections and normalisations can also effectively downweight the residual branch to give better signal propagation properties, but only if the normalisation layer is placed on the residual branch, like in Pre-LN transformers.

Original: Such an effect does not occur for Post-LN transformers, where the normalisation layer is after the residual branch, and we indeed observe in Fig.
Revised: Such an effect does not occur for Post-LN transformers, where the normalisation layer is after the residual branch, and we indeed observe in Fig.

Original: 7 that Post-LN attention-only transformers also suffer from rank collapse at large depths.
Revised: 7 that Post-LN attention-only transformers also suffer from rank collapse at large depths.

Original: This may explain some of the training instabilities of
Revised: This may explain some of the training instabilities of

Original: Post-LN transformers that have been observed in practice (Xiong et al., 2020; Liu et al., 2020).
Revised: Post-LN transformers that have been observed in practice (Xiong et al., 2020; Liu et al., 2020).

Original:  C ONSTRUCTING TRAINABLE DEEP TRANSFORMERS WITHOUT SHORTCUTS
Revised:  C ONSTRUCTING TRAINABLE DEEP TRANSFORMERS WITHOUT SHORTCUTS

Original: To date, the only strategy for rectifying rank collapse in transformers relies on skip/shortcut connections, which “skip” around the trainability issues intrinsic to self-attention layers.
Revised: To date, the only strategy for rectifying rank collapse in transformers relies on skip/shortcut connections, which “skip” around the trainability issues intrinsic to self-attention layers.

Original: We seek instead to tackle this issue directly.
Revised: We seek instead to tackle this issue directly.

Original: To do so, we first develop a better understanding of signal propagation through attention layers, then derive modifications from our insights to achieve faithful signal propagation in deep transformers, allowing them to be trained regardless of the use of skip connections.
Revised: To do so, we first develop a better understanding of signal propagation through attention layers, then derive modifications from our insights to achieve faithful signal propagation in deep transformers, allowing them to be trained regardless of the use of skip connections.

Original: To start with, we consider the simplified setting of a deep attention-only vanilla transformer, and suppose we are in a single-head setting ( h = 1 ) or a multi-head setting where the attention matrix
Revised: To start with, we consider the simplified setting of a deep attention-only vanilla transformer, and suppose we are in a single-head setting ( h = 1 ) or a multi-head setting where the attention matrix

Original: A does not vary across heads.
Revised: A does not vary across heads.

Original: If block l ≤ L has attention matrix A l at initialisation, then the final block’s representation X L takes the following form:
Revised: If block l ≤ L has attention matrix A l at initialisation, then the final block’s representation X L takes the following form:

Original: [Equation]
Revised: [Equation]

Original: where W = (cid:81)
Revised: where W = (cid:81)

Original: Ll =
Revised: Ll =

Original: 1 W
Revised: 1 W

Original: Vl W
Revised: Vl W

Original: Ol ∈ R d × d can be made to be orthogonal at initialisation (so that
Revised: Ol ∈ R d × d can be made to be orthogonal at initialisation (so that

Original: W ⊤ W = I
Revised: W ⊤ W = I

Original: d ), if each W Vl and W Ol are orthogonally initialised (assuming d v = dh ).
Revised: d ), if each W Vl and W Ol are orthogonally initialised (assuming d v = dh ).

Original: Going forward we will assume such an orthogonal initialisation, providing an ablation study in Fig.
Revised: Going forward we will assume such an orthogonal initialisation, providing an ablation study in Fig.

Original: In that case, if we denote by Σ 

Revised: In that case, if we denote by Σ

Original: = X 0 X ⊤ 0 ∈ R T × T the input kernel matrix across locations, and Π l = (cid:81) li =

Revised: = X 0 X ⊤ 0 ∈ R T × T the input kernel matrix across locations, and Π l = (cid:81) li =

Original: A l the product of attention matrices up to the l th block, then the location-wise kernel matrix Σ l = X l X ⊤ l ∈ R T × T at block l simplifies to
Revised: A l the product of attention matrices up to the l th block, then the location-wise kernel matrix Σ l = X l X ⊤ l ∈ R T × T at block l simplifies to

Original: [Equation]
Revised: [Equation]

Original: From this simplified formula for kernel matrices in deep attention-only transformers, we identify three requirements on ( A l ) l :
Revised: From this simplified formula for kernel matrices in deep attention-only transformers, we identify three requirements on ( A l ) l :

Original: (i) Σ l = Π l · Σ 0 · Π ⊤ l must be well-behaved at each block, avoiding degenerate situations such as rank collapse and exploding/vanishing diagonal values.
Revised: (i) Σ l = Π l · Σ 0 · Π ⊤ l must be well-behaved at each block, avoiding degenerate situations such as rank collapse and exploding/vanishing diagonal values.

Original: (ii)
Revised: (ii)

Original: A l must be elementwise non-negative ∀ l (recalling that A l is constructed through the softmax operation Eq.
Revised: A l must be elementwise non-negative ∀ l (recalling that A l is constructed through the softmax operation Eq.

Original: (iii)
Revised: (iii)

Original: A l should be lower triangular ∀ l , for compatibility with causal masked attention.
Revised: A l should be lower triangular ∀ l , for compatibility with causal masked attention.

Original: In Sections 3.1 and 3.2, we focus on finding attention matrices that satisfy our desiderata above, and demonstrate how to modify softmax attention to achieve these attention matrices in Section 3.
Revised: In Sections 3.1 and 3.2, we focus on finding attention matrices that satisfy our desiderata above, and demonstrate how to modify softmax attention to achieve these attention matrices in Section 3.

Original: 3. I DENTITY ATTENTION : ISSUES AND V ALUE -S KIPINIT
Revised: 3. I DENTITY ATTENTION : ISSUES AND V ALUE -S KIPINIT

Original: An obvious solution to our above requirements on ( A l ) l is the trivial one: A l = I T ∀ l , where each sequence location attends only to itself.
Revised: An obvious solution to our above requirements on ( A l ) l is the trivial one: A l = I T ∀ l , where each sequence location attends only to itself.

Original: In this case, Σ L = Σ 0 perfectly preserves the input kernel matrix, and will avoid rank collapse assuming Σ 0 is non-degenerate.
Revised: In this case, Σ L = Σ 0 perfectly preserves the input kernel matrix, and will avoid rank collapse assuming Σ 0 is non-degenerate.

Original: Unfortunately, identity attention isn’t compatible with a viable solution to obtain trainable vanilla transformers.
Revised: Unfortunately, identity attention isn’t compatible with a viable solution to obtain trainable vanilla transformers.

Original: This is because to achieve A l = I we would need to saturate the softmax operation (Eq.
Revised: This is because to achieve A l = I we would need to saturate the softmax operation (Eq.

Original: so that gradients do not pass to the query and key parameters, and the attention matrix stays close to identity during training.
Revised: so that gradients do not pass to the query and key parameters, and the attention matrix stays close to identity during training.

Original: To provide a partial solution to this that achieves identity attention matrix at initialisation yet is still trainable, we introduce our first approach, Value-SkipInit , based on SkipInit (De & Smith, 2020), and the related ReZero method (Bachlechner et al., 2021).
Revised: To provide a partial solution to this that achieves identity attention matrix at initialisation yet is still trainable, we introduce our first approach, Value-SkipInit , based on SkipInit (De & Smith, 2020), and the related ReZero method (Bachlechner et al., 2021).

Original: In Value-SkipInit, we modify the attention operation Attn ( X ) =
Revised: In Value-SkipInit, we modify the attention operation Attn ( X ) =

Original: A ( X
Revised: A ( X

Original: [Equation]
Revised: [Equation]

Original: V ( X ) ,
Revised: V ( X ) ,

Original: to
Revised: to

Original: [Equation]
Revised: [Equation]

Original: with trainable parameters α and β that are initialised to 1 and 0 respectively.
Revised: with trainable parameters α and β that are initialised to 1 and 0 respectively.

Original: Thus, at initialisation, the attention matrix is the identity.
Revised: Thus, at initialisation, the attention matrix is the identity.

Original: For transformers with MLPs blocks, this yields identical behaviour to a standard MLP acting on each sequence location independently at initialisation, so we can apply the DKS or TAT frameworks (Martens et al., 2021; Zhang et al., 2022) to achieve well-behaved signal propagation in the entire model.
Revised: For transformers with MLPs blocks, this yields identical behaviour to a standard MLP acting on each sequence location independently at initialisation, so we can apply the DKS or TAT frameworks (Martens et al., 2021; Zhang et al., 2022) to achieve well-behaved signal propagation in the entire model.

Original: We note that Value-Skipinit is an approach to remove the standard skip connections in transformer blocks, Eq.
Revised: We note that Value-Skipinit is an approach to remove the standard skip connections in transformer blocks, Eq.

Original: (1), but can be construed as adding a skip connection around the value computation, and in that respect isn’t strictly “skipless”.
Revised: (1), but can be construed as adding a skip connection around the value computation, and in that respect isn’t strictly “skipless”.

Original: Moreover, there is useful information contained in the positions of sequence locations that we do not make use of when all attention matrices are identity at initialisation.
Revised: Moreover, there is useful information contained in the positions of sequence locations that we do not make use of when all attention matrices are identity at initialisation.

Original: As a result, we treat Value-Skipinit as a baseline for our main two methods, which we describe next.
Revised: As a result, we treat Value-Skipinit as a baseline for our main two methods, which we describe next.

Original: 3. S IGNAL P RESERVING A TTENTION METHODS
Revised: 3. S IGNAL P RESERVING A TTENTION METHODS

Original: Returning to our requirements on ( A l ) l in Section 3, we see that controlling their product is key to achieving faithful signal propagation.
Revised: Returning to our requirements on ( A l ) l in Section 3, we see that controlling their product is key to achieving faithful signal propagation.

Original: Given that we are now interested in non-identity ( A l ) l , this becomes more difficult.
Revised: Given that we are now interested in non-identity ( A l ) l , this becomes more difficult.

Original: To overcome this, we consider A l of the form
Revised: To overcome this, we consider A l of the form

Original: [Equation]
Revised: [Equation]

Original: such that individual L i ’s cancel in the product, giving Π l =
Revised: such that individual L i ’s cancel in the product, giving Π l =

Original: L l L − 1 0 .
Revised: L l L − 1 0 .

Original: Then if L 0 satisfies
Revised: Then if L 0 satisfies

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: T , we thus have
Revised: T , we thus have

Original: [Equation]
Revised: [Equation]

Original: Assuming input embeddings are initialised independently, and no repeated tokens in the input sequence, we have Σ 

Revised: Assuming input embeddings are initialised independently, and no repeated tokens in the input sequence, we have Σ

Original: = I T in the large width limit, 4 and can thus take L 

Revised: = I T in the large width limit, 4 and can thus take L

Original: = I T .
Revised: = I T .

Original: In practice, we will apply slight modifications to our methods to account for repeated tokens, as detailed in Appendix B.
Revised: In practice, we will apply slight modifications to our methods to account for repeated tokens, as detailed in Appendix B.

Original: Now if L l is lower triangular, then from Eq.
Revised: Now if L l is lower triangular, then from Eq.

Original: (9) we see it is simply a Cholesky factor for the kernel matrix Σ l .
Revised: (9) we see it is simply a Cholesky factor for the kernel matrix Σ l .

Original: By the uniqueness of Cholesky factors for PSD matrices up to sign, this means we just need to choose { Σ l } Ll =0 to be a family of well-behaved kernel matrices that satisfy 5 our nonnegativity constraint (ii) on A l =
Revised: By the uniqueness of Cholesky factors for PSD matrices up to sign, this means we just need to choose { Σ l } Ll =0 to be a family of well-behaved kernel matrices that satisfy 5 our nonnegativity constraint (ii) on A l =

Original: L l L − 1 l − 1 .
Revised: L l L − 1 l − 1 .

Original: We identify two such families, which give rise to our main method, Signal Preserving Attention (SPA):
Revised: We identify two such families, which give rise to our main method, Signal Preserving Attention (SPA):

Original: Uniform (U-SPA): Σ l ( ρ l ) =
Revised: Uniform (U-SPA): Σ l ( ρ l ) =

Original: (1 − ρ l ) I T + ρ l 11 ⊤
Revised: (1 − ρ l ) I T + ρ l 11 ⊤

Original: Here, the kernel matrix
Revised: Here, the kernel matrix

Original: Σ
Revised: Σ

Original: l
Revised: l

Original: ( ρ l
Revised: ( ρ l

Original: ) has diagonal entries
Revised: ) has diagonal entries

Original: equal to 1 and off-
Revised: equal to 1 and off-

Original: diagonal
Revised: diagonal

Original: entries
Revised: entries

Original: equal
Revised: equal

Original: to
Revised: to

Original: ρ l .
Revised: ρ l .

Original: The condition ρ l ≤ ρ
Revised: The condition ρ l ≤ ρ

Original: l +
Revised: l +

Original: is
Revised: is

Original: required for elementwise non-
Revised: required for elementwise non-

Original: negativity
Revised: negativity

Original: of the A ’s, as shown in Theorem 1.
Revised: of the A ’s, as shown in Theorem 1.

Original: Setting ρ 0 = 0 yields identity input kernel matrix, and as
Revised: Setting ρ 0 = 0 yields identity input kernel matrix, and as

Original: long
Revised: long

Original: as ρ L < 1 , we avoid rank collapse in the
Revised: as ρ L < 1 , we avoid rank collapse in the

Original: skipless
Revised: skipless

Original: attention-only setting.
Revised: attention-only setting.

Original: Exponential (E-SPA): (cid:0) Σ l ( γ l ) (cid:1) i,j = exp ( − γ l | i − j | ) .
Revised: Exponential (E-SPA): (cid:0) Σ l ( γ l ) (cid:1) i,j = exp ( − γ l | i − j | ) .

Original: Here, diagonal entries are again 1, but now off-diagonals decay exponentially for more distant locations, with decay rate γ l .
Revised: Here, diagonal entries are again 1, but now off-diagonals decay exponentially for more distant locations, with decay rate γ l .

Original: Thus, unlike
Revised: Thus, unlike

Original: U-SPA, E-SPA captures the notion of positional encoding , as the vector representations for nearby locations have larger inner products (i.e. are more similar to each other).
Revised: U-SPA, E-SPA captures the notion of positional encoding , as the vector representations for nearby locations have larger inner products (i.e. are more similar to each other).

Original: The condition γ l ≥ γ l + is required for elementwise non-negativity of the A ’s, as established in Theorem 2.
Revised: The condition γ l ≥ γ l + is required for elementwise non-negativity of the A ’s, as established in Theorem 2.

Original: Setting γ 0 = ∞ yields the identity input kernel matrix, and rank collapse will be prevented as long as γ L > 0 .
Revised: Setting γ 0 = ∞ yields the identity input kernel matrix, and rank collapse will be prevented as long as γ L > 0 .

Original: We state and prove Theorems 1 and 2 in Appendix I, and in particular provide a closed-form solution for L l L − 1 l − 1 in the case of E-SPA in Theorem 3, enabling cheap computation.
Revised: We state and prove Theorems 1 and 2 in Appendix I, and in particular provide a closed-form solution for L l L − 1 l − 1 in the case of E-SPA in Theorem 3, enabling cheap computation.

Original: From these theorems, we see that our proposed SPA approaches are viable as long as the kernel matrix values become progressively larger (in an elementwise fashion) as depth increases, as dictated by ( ρ l ) Ll =1 and ( γ l ) Ll =1 .
Revised: From these theorems, we see that our proposed SPA approaches are viable as long as the kernel matrix values become progressively larger (in an elementwise fashion) as depth increases, as dictated by ( ρ l ) Ll =1 and ( γ l ) Ll =1 .

Original: We find ρ L = 0 .
Revised: We find ρ L = 0 .

Original: 8 and γ L = 0 .
Revised: 8 and γ L = 0 .

Original: 005 to be good default choices for the final block, and describe how to vary ρ l and γ l across depth in Appendix C. In Alg. 1 in Appendix D, we summarise how to construct a trainable self-attention layer with our main method E-SPA using ideas from Section 3.3, given an input-output decay rate pair γ in , γ out (in the notation of Alg. 1).
Revised: 005 to be good default choices for the final block, and describe how to vary ρ l and γ l across depth in Appendix C. In Alg. 1 in Appendix D, we summarise how to construct a trainable self-attention layer with our main method E-SPA using ideas from Section 3.3, given an input-output decay rate pair γ in , γ out (in the notation of Alg. 1).

Original: Given a decreasing set of decay rates ( γ l ) Ll =0 , at block l we set γ in = γ l − 1 and γ out = γ l .
Revised: Given a decreasing set of decay rates ( γ l ) Ll =0 , at block l we set γ in = γ l − 1 and γ out = γ l .

Original: In Fig.
Revised: In Fig.

Original: 1, we verify that our two proposed SPA schemes, U-SPA and E-SPA, successfully avoid rank collapse in attention-only vanilla transformers even at large depths.
Revised: 1, we verify that our two proposed SPA schemes, U-SPA and E-SPA, successfully avoid rank collapse in attention-only vanilla transformers even at large depths.

Original: Moreover, because Σ l has diagonals equal to 1 for all l , there is an implicit mechanism in these two schemes to control the representation vector norms across all sequence locations at deep layers.
Revised: Moreover, because Σ l has diagonals equal to 1 for all l , there is an implicit mechanism in these two schemes to control the representation vector norms across all sequence locations at deep layers.

Original: This means that they can be used with or without normalisation layers, as we will verify empirically in Section 4.
Revised: This means that they can be used with or without normalisation layers, as we will verify empirically in Section 4.

Original: Moreover, in Fig.
Revised: Moreover, in Fig.

Original: 1 we see that E-SPA observes a recency bias as expected, where representation vectors for nearby locations have larger cosine similarity, as seen with positional encoding schemes like ALiBi (Press et al., 2022)
Revised: 1 we see that E-SPA observes a recency bias as expected, where representation vectors for nearby locations have larger cosine similarity, as seen with positional encoding schemes like ALiBi (Press et al., 2022)

Original: (Fig. 7).
Revised: (Fig. 7).

Original: As a result, even though all three of our approaches successfully avoid rank collapse, we expect E-SPA to outperform both U-SPA and Value-SkipInit.
Revised: As a result, even though all three of our approaches successfully avoid rank collapse, we expect E-SPA to outperform both U-SPA and Value-SkipInit.

Original: 3. R EVERSE ENGINEERING SELF - ATTENTION LAYERS AT INITIALISATION
Revised: 3. R EVERSE ENGINEERING SELF - ATTENTION LAYERS AT INITIALISATION

Original: In Section 3.2 we identified two families of lower-triangular non-negative attention matrices ( A l ) l which enable us to obtain well-behaved kernel matrices Σ L at large depth L , and hence faithful signal propagation.
Revised: In Section 3.2 we identified two families of lower-triangular non-negative attention matrices ( A l ) l which enable us to obtain well-behaved kernel matrices Σ L at large depth L , and hence faithful signal propagation.

Original: It remains to show how we can actually realise these attention matrices via parameter initialisation and minor modifications of the attention mechanism.
Revised: It remains to show how we can actually realise these attention matrices via parameter initialisation and minor modifications of the attention mechanism.

Original: More precisely, we will show how for any given lower-triangular A ∈ R T × T with non-negative entries, the masked softmax self-attention layer Eq.
Revised: More precisely, we will show how for any given lower-triangular A ∈ R T × T with non-negative entries, the masked softmax self-attention layer Eq.

Original: (4) can be initialised and augmented such that its output is exactly Attn ( X )
Revised: (4) can be initialised and augmented such that its output is exactly Attn ( X )

Original: =
Revised: =

Original: AV ( X ) at initialisation.
Revised: AV ( X ) at initialisation.

Original: To do so, we first define matrices D , P ∈ R T × T such that A = DP , where D is diagonal with positive entries and P is lower triangular with row sums 1.
Revised: To do so, we first define matrices D , P ∈ R T × T such that A = DP , where D is diagonal with positive entries and P is lower triangular with row sums 1.

Original: Then if B = log ( P ) and M is the causal mask M i,j = 1 { i ≥ j } , we set
Revised: Then if B = log ( P ) and M is the causal mask M i,j = 1 { i ≥ j } , we set

Original: [Equation]
Revised: [Equation]

Original: which reduces to Attn ( X )
Revised: which reduces to Attn ( X )

Original: =
Revised: =

Original: AV ( X ) (with a data-independent attention matrix) at initialisation if the query-key dot product 1 √ d k Q ( X ) K ( X ) ⊤ = 0 .
Revised: AV ( X ) (with a data-independent attention matrix) at initialisation if the query-key dot product 1 √ d k Q ( X ) K ( X ) ⊤ = 0 .

Original: We note that zero query-key dot products occur if one uses a 1 d k scaling rather 1 √ d k in the infinite width limit for independently initialised W Q , W K
Revised: We note that zero query-key dot products occur if one uses a 1 d k scaling rather 1 √ d k in the infinite width limit for independently initialised W Q , W K

Original: (Yang, 2019).
Revised: (Yang, 2019).

Original: In practice, to achieve zero initial query-key dot product, we can initialise either: 1) W Q = 0 , 2) W K = 0 , or 3) both W Q and W K to have small initial scale (which achieves approximately zero dot product).
Revised: In practice, to achieve zero initial query-key dot product, we can initialise either: 1) W Q = 0 , 2) W K = 0 , or 3) both W Q and W K to have small initial scale (which achieves approximately zero dot product).

Original: In our experiments we found these three options to all perform similarly, and decided to use option 1): W Q = 0 .
Revised: In our experiments we found these three options to all perform similarly, and decided to use option 1): W Q = 0 .

Original: An empirical evaluation of the sensitivity to the choice of initial scale in option 3) is provided in Fig. 8.
Revised: An empirical evaluation of the sensitivity to the choice of initial scale in option 3) is provided in Fig. 8.

Original: In Eq.
Revised: In Eq.

Original: (10), D acts as a non-trainable location-dependent rescaling , and is needed to realise arbitrary attention matrices since the softmax output P ( X ) is constrained to have row sums equal to 1.
Revised: (10), D acts as a non-trainable location-dependent rescaling , and is needed to realise arbitrary attention matrices since the softmax output P ( X ) is constrained to have row sums equal to 1.

Original: 7 Given target kernel matrices with 1’s on the diagonal (which we have in SPA) its inclusion is akin to using
Revised: 7 Given target kernel matrices with 1’s on the diagonal (which we have in SPA) its inclusion is akin to using

Original: RMSNorm at initialisation.
Revised: RMSNorm at initialisation.

Original: However, this will gradually stop being true during training, leading to a (slightly) different model class.
Revised: However, this will gradually stop being true during training, leading to a (slightly) different model class.

Original: The additive pre-softmax biases B will also have an effect on the model class, but this will be similar to that of the ALiBi positional encoder (Press et al., 2022), which too involves a non-trainable bias matrix being added to the logits.
Revised: The additive pre-softmax biases B will also have an effect on the model class, but this will be similar to that of the ALiBi positional encoder (Press et al., 2022), which too involves a non-trainable bias matrix being added to the logits.

Original: In our early experiments we tried including a trainable gain parameter on the bias matrix, initialised to 1, in order to better preserve the model class but found that this didn’t have a significant impact on training performance.
Revised: In our early experiments we tried including a trainable gain parameter on the bias matrix, initialised to 1, in order to better preserve the model class but found that this didn’t have a significant impact on training performance.

Original: We also note that this approach to controlling the attention matrix by zeroing the query-key dot product at initialisation is compatible with popular positional encodings like Relative (Shaw et al., 2018; Huang et al., 2019; Dai et al., 2019) and RoPE (Su et al., 2021), as discussed in Appendix E.
Revised: We also note that this approach to controlling the attention matrix by zeroing the query-key dot product at initialisation is compatible with popular positional encodings like Relative (Shaw et al., 2018; Huang et al., 2019; Dai et al., 2019) and RoPE (Su et al., 2021), as discussed in Appendix E.

Original: Unless stated otherwise, we use RoPE in our experiments, and provide an ablation in Fig. 10.
Revised: Unless stated otherwise, we use RoPE in our experiments, and provide an ablation in Fig. 10.

Original: 3. A DDRESSING MLP BLOCKS AND SKIP CONNECTIONS
Revised: 3. A DDRESSING MLP BLOCKS AND SKIP CONNECTIONS

Original: Up until this point we have restricted our focus to attention-only skipless transformers, for the sake of simplicity.
Revised: Up until this point we have restricted our focus to attention-only skipless transformers, for the sake of simplicity.

Original: However, MLP blocks are an important part of the transformer architecture that we must also address.
Revised: However, MLP blocks are an important part of the transformer architecture that we must also address.

Original: And we would like for our approaches to be compatible with skip connections too, both for the sake of generality, and because they might combine profitably.
Revised: And we would like for our approaches to be compatible with skip connections too, both for the sake of generality, and because they might combine profitably.

Original: MLP blocks Because MLP blocks operate on the representation vectors independently across locations, their effect on the kernel matrix can be easily computed using the standard limiting kernel formulas for MLPs (Neal, 2012) which are exact in the infinite width limit.
Revised: MLP blocks Because MLP blocks operate on the representation vectors independently across locations, their effect on the kernel matrix can be easily computed using the standard limiting kernel formulas for MLPs (Neal, 2012) which are exact in the infinite width limit.

Original: In particular, there is a known f that maps the kernel matrix Σ l of an MLP block’s input sequence to the kernel matrix ˆΣ l of its output sequence.
Revised: In particular, there is a known f that maps the kernel matrix Σ l of an MLP block’s input sequence to the kernel matrix ˆΣ l of its output sequence.

Original: In principle, one could modify SPA to account for this change to the kernel matrix by taking A l =
Revised: In principle, one could modify SPA to account for this change to the kernel matrix by taking A l =

Original: L l ˆ L − 1 l − 1 , where L l and ˆ L l are the Cholesky factors of Σ l and ˆΣ l respectively.
Revised: L l ˆ L − 1 l − 1 , where L l and ˆ L l are the Cholesky factors of Σ l and ˆΣ l respectively.

Original: Unfortunately, there is no guarantee that the resulting A l would be elementwise non-negative in general.
Revised: Unfortunately, there is no guarantee that the resulting A l would be elementwise non-negative in general.

Original: In our experiments we thus elected to ignore the effect on the kernel matrices of the MLP blocks, approximating f as the identity function, which we found to work well enough in practice.
Revised: In our experiments we thus elected to ignore the effect on the kernel matrices of the MLP blocks, approximating f as the identity function, which we found to work well enough in practice.

Original: As discussed in Martens et al.
Revised: As discussed in Martens et al.

Original: (2021), this approximation becomes better as the MLP blocks tend to linear functions (for which f is exactly identity), which will happen as we increase the shortcut weight α relative to the residual weight β
Revised: (2021), this approximation becomes better as the MLP blocks tend to linear functions (for which f is exactly identity), which will happen as we increase the shortcut weight α relative to the residual weight β

Original: (see Eq.
Revised: (see Eq.

Original: (1)), or when the MLP’s activation functions are close to linear.
Revised: (1)), or when the MLP’s activation functions are close to linear.

Original: Notably, the latter condition will tend to be true when using DKS or TAT to transform the activation functions in deep networks.
Revised: Notably, the latter condition will tend to be true when using DKS or TAT to transform the activation functions in deep networks.

Original: Note, when using DKS, f decreases the size of off-diagonal elements of the kernel matrix, which intuitively will help to combat rank collapse.
Revised: Note, when using DKS, f decreases the size of off-diagonal elements of the kernel matrix, which intuitively will help to combat rank collapse.

Original: Skip connections When using skip connections as in Eq.
Revised: Skip connections When using skip connections as in Eq.

Original: (1), the output kernel matrix of a block is given by Σ block =
Revised: (1), the output kernel matrix of a block is given by Σ block =

Original: α 2 Σ shortcut + β 2 · Σ residual .
Revised: α 2 Σ shortcut + β 2 · Σ residual .

Original: One of the main ways that kernel matrices degenerate in
Revised: One of the main ways that kernel matrices degenerate in

Original: DNNs is when their diagonal entries either explode or shrink with depth.
Revised: DNNs is when their diagonal entries either explode or shrink with depth.

Original: As shown by Martens et al.
Revised: As shown by Martens et al.

Original: (2021), this can be prevented in fully-connected and convolutional DNN architectures by rescaling the output of the activation functions (which happens automatically as part of DKS or TAT), and by replacing weighted sums with “normalised sums”, which yields normalised
Revised: (2021), this can be prevented in fully-connected and convolutional DNN architectures by rescaling the output of the activation functions (which happens automatically as part of DKS or TAT), and by replacing weighted sums with “normalised sums”, which yields normalised

Original: skip connections defined by the condition α 2 + β 2 = 1 .
Revised: skip connections defined by the condition α 2 + β 2 = 1 .

Original: When using normalised skip connections with U-SPA, both Σ shortcut and Σ residual will have the form Σ( ρ ) =
Revised: When using normalised skip connections with U-SPA, both Σ shortcut and Σ residual will have the form Σ( ρ ) =

Original: (1 − ρ ) I T + ρ 11 ⊤ for some ρ , and thus so will Σ block .
Revised: (1 − ρ ) I T + ρ 11 ⊤ for some ρ , and thus so will Σ block .

Original: Moreover, we will have ρ block =
Revised: Moreover, we will have ρ block =

Original: α 2 ρ shortcut + β 2 · ρ residual , which is less than ρ residual .
Revised: α 2 ρ shortcut + β 2 · ρ residual , which is less than ρ residual .

Original: This means we can easily adjust U-SPA to be compatible with normalised skip connections by replacing L l in the formula A l =
Revised: This means we can easily adjust U-SPA to be compatible with normalised skip connections by replacing L l in the formula A l =

Original: L l L − 1 l − 1 with the Cholesky factor of Σ(( ρ l − α 2 ρ l − 1 ) /β 2 ) .
Revised: L l L − 1 l − 1 with the Cholesky factor of Σ(( ρ l − α 2 ρ l − 1 ) /β 2 ) .

Original: In the case of E-SPA, we note that Σ block won’t be of the form (cid:0) Σ( γ ) (cid:1) i,j = exp ( − γ | i − j | ) for some γ , even when both Σ shortcut and Σ residual are.
Revised: In the case of E-SPA, we note that Σ block won’t be of the form (cid:0) Σ( γ ) (cid:1) i,j = exp ( − γ | i − j | ) for some γ , even when both Σ shortcut and Σ residual are.

Original: To work around this, we use an approximation described in Appendix F, which seeks to make the combined effect on the kernel matrix of the normalised skip and attention block approximately equal to the effect of a skipless attention block.
Revised: To work around this, we use an approximation described in Appendix F, which seeks to make the combined effect on the kernel matrix of the normalised skip and attention block approximately equal to the effect of a skipless attention block.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now assess the capabilities of our proposed methods in training deep skipless and/or normaliserfree transformers.
Revised: We now assess the capabilities of our proposed methods in training deep skipless and/or normaliserfree transformers.

Original: Our main experiment setting uses a transformer with 36 transformer blocks, which is deep enough that the effects of poor signal propagation and rank collapse render skipless training without modifications impossible.
Revised: Our main experiment setting uses a transformer with 36 transformer blocks, which is deep enough that the effects of poor signal propagation and rank collapse render skipless training without modifications impossible.

Original: We begin our investigation on WikiText-103 (Merity et al., 2017) focusing primarily on training performance of our various methods, before moving to the larger C dataset (Raffel et al., 2019), where overfitting is not an issue.
Revised: We begin our investigation on WikiText-103 (Merity et al., 2017) focusing primarily on training performance of our various methods, before moving to the larger C dataset (Raffel et al., 2019), where overfitting is not an issue.

Original: We use Adam optimiser (Kingma & Ba, 2014), as is standard for transformers, and train without dropout (Srivastava et al., 2014).
Revised: We use Adam optimiser (Kingma & Ba, 2014), as is standard for transformers, and train without dropout (Srivastava et al., 2014).

Original: Additional results and experimental details are provided in Appendices G and H respectively.
Revised: Additional results and experimental details are provided in Appendices G and H respectively.

Original: WikiText-103 baselines To start with, we verify that a standard deep transformer without skip connections is untrainable even with normalisation layers (LN) and transformed activations, and that our approaches remedy this.
Revised: WikiText-103 baselines To start with, we verify that a standard deep transformer without skip connections is untrainable even with normalisation layers (LN) and transformed activations, and that our approaches remedy this.

Original: Fig.
Revised: Fig.

Original: 2 compares vanilla transformers using our proposed SPA
Revised: 2 compares vanilla transformers using our proposed SPA

Original: [Figure]
Revised: [Figure]

Original: methods and Value-Skipinit to standard transformers both with and without skips, on a 36 block transformer.
Revised: methods and Value-Skipinit to standard transformers both with and without skips, on a 36 block transformer.

Original: We clearly see that removing skip connections from a standard transformer makes it untrainable, with training loss plateauing around 7.5.
Revised: We clearly see that removing skip connections from a standard transformer makes it untrainable, with training loss plateauing around 7.5.

Original: This holds true too even when we use DKS to transform the GeLU MLPs, highlighting that the issue lies with the attention layers, which suffer from rank collapse as shown in Fig.
Revised: This holds true too even when we use DKS to transform the GeLU MLPs, highlighting that the issue lies with the attention layers, which suffer from rank collapse as shown in Fig.

Original: [Figure]
Revised: [Figure]

Original: On the other hand, all three of our approaches train even for vanilla deep transformers, with our E-SPA method outperforming U-SPA and Value-Skipinit.
Revised: On the other hand, all three of our approaches train even for vanilla deep transformers, with our E-SPA method outperforming U-SPA and Value-Skipinit.

Original: However, the default transformer with skips and
Revised: However, the default transformer with skips and

Original: LN still retains a training speed advantage compared to our skipless methods, mirroring the situation for CNNs without powerful second-order optimisers (Martens et al., 2021; Zhang et al., 2022).
Revised: LN still retains a training speed advantage compared to our skipless methods, mirroring the situation for CNNs without powerful second-order optimisers (Martens et al., 2021; Zhang et al., 2022).

Original: In Table
Revised: In Table

Original: [Figure]
Revised: [Figure]

Original: we assess the effect of different activation functions in the MLP blocks, as well as the use of LN, in skipless transformers using our proposed methods.
Revised: we assess the effect of different activation functions in the MLP blocks, as well as the use of LN, in skipless transformers using our proposed methods.

Original: We see that at depth 36 we achieve good training performance for a range of activations:
Revised: We see that at depth 36 we achieve good training performance for a range of activations:

Original: DKS-transformed GeLU, TAT-transformed Leaky
Revised: DKS-transformed GeLU, TAT-transformed Leaky

Original: ReLU, as well untransformed GeLU (Hendrycks &
Revised: ReLU, as well untransformed GeLU (Hendrycks &

Original: Gimpel, 2016), but not untransformed Sigmoid.
Revised: Gimpel, 2016), but not untransformed Sigmoid.

Original: We also see that layer normalisation is relatively unimportant for training speed, and can even be harmful with transformed activations when using
Revised: We also see that layer normalisation is relatively unimportant for training speed, and can even be harmful with transformed activations when using

Original: SPA, which already has an inbuilt mechanism to control activation norms (as discussed at the end of
Revised: SPA, which already has an inbuilt mechanism to control activation norms (as discussed at the end of

Original: Section 3.2).
Revised: Section 3.2).

Original: [Table]
Revised: [Table]

Original: Normalised skip connections In Fig.
Revised: Normalised skip connections In Fig.

Original: 3, we see that one way to match the training loss of the default transformer, without more iterations, is by using normalised skip connections.
Revised: 3, we see that one way to match the training loss of the default transformer, without more iterations, is by using normalised skip connections.

Original: While this is perhaps unsurprising, we observe that our E-SPA method (left) matches standard training both with and without normalisation, whereas a standard Transformer with normalised skip connections (right) requires normalisation in order to match the training speed of the default Pre-LN.
Revised: While this is perhaps unsurprising, we observe that our E-SPA method (left) matches standard training both with and without normalisation, whereas a standard Transformer with normalised skip connections (right) requires normalisation in order to match the training speed of the default Pre-LN.

Original: C4 baseline So far, we tested our proposed methods on WikiText-103 (Merity et al., 2017), on which we observed overfitting without the use of extra regularisation.
Revised: C4 baseline So far, we tested our proposed methods on WikiText-103 (Merity et al., 2017), on which we observed overfitting without the use of extra regularisation.

Original: Therefore, we further compare our methods to standard transformers on a larger dataset, C4 (Raffel et al., 2019), where overfitting isn’t an issue.
Revised: Therefore, we further compare our methods to standard transformers on a larger dataset, C4 (Raffel et al., 2019), where overfitting isn’t an issue.

Original: Importantly, we see similar trends across validation 8 (Fig. 4a), training (Fig. 13) and downstream task (Table 6) performance, and so the benefits of our methods do extend beyond training.
Revised: Importantly, we see similar trends across validation 8 (Fig. 4a), training (Fig. 13) and downstream task (Table 6) performance, and so the benefits of our methods do extend beyond training.

Original: Due to the memory
Revised: Due to the memory

Original: [Figure]
Revised: [Figure]

Original: overhead of longer sequences, we use a 32-block transformer.
Revised: overhead of longer sequences, we use a 32-block transformer.

Original: One can notice that E-SPA performs the best among skipless transformers on all settings: training, validation and downstream tasks.
Revised: One can notice that E-SPA performs the best among skipless transformers on all settings: training, validation and downstream tasks.

Original: [Table]
Revised: [Table]

Original: Moreover, in Table 2 we find that E-SPA with normalised skips and LN outperforms the default PreLN transformer, achieving 24.0 (vs 24.7) validation perplexity on C4 after 50K steps.
Revised: Moreover, in Table 2 we find that E-SPA with normalised skips and LN outperforms the default PreLN transformer, achieving 24.0 (vs 24.7) validation perplexity on C4 after 50K steps.

Original: Even without LN,
Revised: Even without LN,

Original: E-SPA exactly matches the Pre-LN transformer in training speed, and outperforms a range of baselines designed to remove LN, including Stable ResNet (Hayou et al., 2021;
Revised: E-SPA exactly matches the Pre-LN transformer in training speed, and outperforms a range of baselines designed to remove LN, including Stable ResNet (Hayou et al., 2021;

Original: Noci et al., 2022) and SkipInit (De & Smith, 2020), both with & without LN.
Revised: Noci et al., 2022) and SkipInit (De & Smith, 2020), both with & without LN.

Original: Longer training matches default transformers In Figs. 2 and 4a, we observed that while our methods are able to train deep skipless transformers (a result which is unprecedented in the literature), there is a significant gap in training speed compared to a standard Pre-LN transformer (with skips).
Revised: Longer training matches default transformers In Figs. 2 and 4a, we observed that while our methods are able to train deep skipless transformers (a result which is unprecedented in the literature), there is a significant gap in training speed compared to a standard Pre-LN transformer (with skips).

Original: Martens et al.
Revised: Martens et al.

Original: (2021) and Zhang et al.
Revised: (2021) and Zhang et al.

Original: (2022) observed a similar training speed gap for skipless CNNs, and showed that such a gap can be closed by using more sophisticated second order optimisers like K-FAC (Martens & Grosse, 2015) or Shampoo (Gupta et al., 2018).
Revised: (2022) observed a similar training speed gap for skipless CNNs, and showed that such a gap can be closed by using more sophisticated second order optimisers like K-FAC (Martens & Grosse, 2015) or Shampoo (Gupta et al., 2018).

Original: As second order optimisers for transformers are not well established, we instead demonstrate that the training loss gap can be closed by simply training for longer in Fig. 4b.
Revised: As second order optimisers for transformers are not well established, we instead demonstrate that the training loss gap can be closed by simply training for longer in Fig. 4b.

Original: We observe that our E-SPA method matches the training loss of a standard pre-LN transformer on C4 if one trains for around 5 times longer with Adam, in line with the findings from the convolutional case (Zhang et al., 2022).
Revised: We observe that our E-SPA method matches the training loss of a standard pre-LN transformer on C4 if one trains for around 5 times longer with Adam, in line with the findings from the convolutional case (Zhang et al., 2022).

Original: An equivalent plot for WikiText-103 is provided in Fig.
Revised: An equivalent plot for WikiText-103 is provided in Fig.

Original: [Table]
Revised: [Table]

Original: Depth Scaling Finally, as unmodified networks suffer from worse signal propagation properties at larger depths, it is natural to ask how our modified vanilla transformers perform as depth increases.
Revised: Depth Scaling Finally, as unmodified networks suffer from worse signal propagation properties at larger depths, it is natural to ask how our modified vanilla transformers perform as depth increases.

Original: In Table 3, we compare the training performance of different depths (36, 72, and 108) at a range of training step budgets on WikiText-103.
Revised: In Table 3, we compare the training performance of different depths (36, 72, and 108) at a range of training step budgets on WikiText-103.

Original: We find that whilst the shallower depth 36 network trains fastest initially over 100K steps, it is matched at 400K steps and then surpassed at 1000K steps by the larger capacity depth 72 network.
Revised: We find that whilst the shallower depth 36 network trains fastest initially over 100K steps, it is matched at 400K steps and then surpassed at 1000K steps by the larger capacity depth 72 network.

Original: Moreover, our depth 108 vanilla transformer is able to close the gap in performance to its depth 36 counterpart with longer training, going from 0.
Revised: Moreover, our depth 108 vanilla transformer is able to close the gap in performance to its depth 36 counterpart with longer training, going from 0.

Original: at 100K steps to just 0.01 at 1000K steps.
Revised: at 100K steps to just 0.01 at 1000K steps.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown for the first time that it is possible to successfully train deep transformers without skip connections or normalisation layers.
Revised: We have shown for the first time that it is possible to successfully train deep transformers without skip connections or normalisation layers.

Original: To do so, we have proposed 3 approaches: E-SPA, U-SPA and Value-Skipinit, each of which control the attention matrices of a transformer to enable faithful signal propagation even at large depths.
Revised: To do so, we have proposed 3 approaches: E-SPA, U-SPA and Value-Skipinit, each of which control the attention matrices of a transformer to enable faithful signal propagation even at large depths.

Original: Our best approach, E-SPA enables deep vanilla transformers to match their standard counterparts with around 5 times more iterations, and also deep transformers without normalisation to match the training speed of standard ones.
Revised: Our best approach, E-SPA enables deep vanilla transformers to match their standard counterparts with around 5 times more iterations, and also deep transformers without normalisation to match the training speed of standard ones.

Original: We hope that our work may potentially pave the way to new and improved architectures, and more research into improving the capabilities of deep learning in practice using insights from theory.
Revised: We hope that our work may potentially pave the way to new and improved architectures, and more research into improving the capabilities of deep learning in practice using insights from theory.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Pseudocode for our main approach, E-SPA, can be found in Alg. 1, using the notation and setup provided in Section 2.
Revised: Pseudocode for our main approach, E-SPA, can be found in Alg. 1, using the notation and setup provided in Section 2.

Original: All experimental details can be found in Appendix H, including general and experiment-specific implementation details.
Revised: All experimental details can be found in Appendix H, including general and experiment-specific implementation details.

Original: A C OMPATIBILITY WITH NON - CAUSAL ATTENTION
Revised: A C OMPATIBILITY WITH NON - CAUSAL ATTENTION

Original: In Section 3, we focus on causal masked self-attention for two reasons.
Revised: In Section 3, we focus on causal masked self-attention for two reasons.

Original: First, next-token prediction using causal masked self-attention is arguably the most popular setting for self-attention.
Revised: First, next-token prediction using causal masked self-attention is arguably the most popular setting for self-attention.

Original: And second, it is a more challenging setting to work with in terms of controlling signal propagation, due to the additional constraint that attention matrices must be lower triangular.
Revised: And second, it is a more challenging setting to work with in terms of controlling signal propagation, due to the additional constraint that attention matrices must be lower triangular.

Original: In this section we describe how our methods can be made compatible with non-causal masked attention, where the attention matrices are no longer required to be lower triangular.
Revised: In this section we describe how our methods can be made compatible with non-causal masked attention, where the attention matrices are no longer required to be lower triangular.

Original: To start with, Value-SkipInit does not modify the softmax-attention computation and hence is already compatible with any form of attention.
Revised: To start with, Value-SkipInit does not modify the softmax-attention computation and hence is already compatible with any form of attention.

Original: For our SPA methods, it is straightforward to extend to non-causal attention by changing L l in Eqs.
Revised: For our SPA methods, it is straightforward to extend to non-causal attention by changing L l in Eqs.

Original: (8) and (9) from being the Cholesky decomposition of Σ l to being the (symmetric) matrix square root of Σ l .
Revised: (8) and (9) from being the Cholesky decomposition of Σ l to being the (symmetric) matrix square root of Σ l .

Original: In this case, for U-SPA it is possible to analytically calculate that A l in Eq.
Revised: In this case, for U-SPA it is possible to analytically calculate that A l in Eq.

Original: (8) will be element-wise non-negative if ρ l ≥ ρ l − 1 (exactly like the Cholesky case in Theorem 1).
Revised: (8) will be element-wise non-negative if ρ l ≥ ρ l − 1 (exactly like the Cholesky case in Theorem 1).

Original: This is easy to see because the matrix square-root, inverses and products of uniform kernel matrices are all still uniform of the form Σ( ρ ) =
Revised: This is easy to see because the matrix square-root, inverses and products of uniform kernel matrices are all still uniform of the form Σ( ρ ) =

Original: (1 − ρ ) I T + ρ 11 ⊤ (up to positive rescaling), so that A l will be too, and one simply needs to track ρ and verify that it is positive.
Revised: (1 − ρ ) I T + ρ 11 ⊤ (up to positive rescaling), so that A l will be too, and one simply needs to track ρ and verify that it is positive.

Original: For E-SPA we have verified empirically that the resulting A l = L l L − 1 l − 1 will be non-negative if γ l ≤ γ l − 1 , just like the Cholesky case in Theorem 2.
Revised: For E-SPA we have verified empirically that the resulting A l = L l L − 1 l − 1 will be non-negative if γ l ≤ γ l − 1 , just like the Cholesky case in Theorem 2.

Original: B M ODIFICATIONS TO SPA METHODS FOR REPEATED TOKENS
Revised: B M ODIFICATIONS TO SPA METHODS FOR REPEATED TOKENS

Original: For simplicity, we assumed that our input sequences had no repeated tokens when presenting SPA in Section 3.
Revised: For simplicity, we assumed that our input sequences had no repeated tokens when presenting SPA in Section 3.

Original: This meant that we could take the input kernel matrix Σ 0 to be the identity, with zero off-diagonals, which was convenient for our construction of SPA.
Revised: This meant that we could take the input kernel matrix Σ 0 to be the identity, with zero off-diagonals, which was convenient for our construction of SPA.

Original: The effect of repeated tokens, e.g. if the word ‘cat’ occurs multiple times in the same sentence, is that our input kernel matrices Σwill have non-zero off-diagonal entries, corresponding to entries where a token is repeated.
Revised: The effect of repeated tokens, e.g. if the word ‘cat’ occurs multiple times in the same sentence, is that our input kernel matrices Σwill have non-zero off-diagonal entries, corresponding to entries where a token is repeated.

Original: This will impact the kernel matrices Σ l at deeper layers with SPA, particularly the diagonal values of Σ l , which we would like to able to control.
Revised: This will impact the kernel matrices Σ l at deeper layers with SPA, particularly the diagonal values of Σ l , which we would like to able to control.

Original: In this section we discuss how we can modify our SPA approaches to account for the fact that we will often be working with sequences where a fraction of the tokens are repeated.
Revised: In this section we discuss how we can modify our SPA approaches to account for the fact that we will often be working with sequences where a fraction of the tokens are repeated.

Original: We stress that the general principle that all our methods (Value-SkipInit and SPA methods) follow is independent of the input kernel matrix (i.e. independent of duplicate input tokens): we seek to prevent the product of attention matrices from deviating away from the identity matrix and degenerating to a rank-1 matrix.
Revised: We stress that the general principle that all our methods (Value-SkipInit and SPA methods) follow is independent of the input kernel matrix (i.e. independent of duplicate input tokens): we seek to prevent the product of attention matrices from deviating away from the identity matrix and degenerating to a rank-1 matrix.

Original: From Eq.
Revised: From Eq.

Original: (6), we see that if the product of attention-matrices is rankthen regardless of the input kernel we will have a rank-1 output kernel i.e. rank collapse (Dong et al., 2021).
Revised: (6), we see that if the product of attention-matrices is rankthen regardless of the input kernel we will have a rank-1 output kernel i.e. rank collapse (Dong et al., 2021).

Original: On the other hand, if we control the deviation of the attention matrix product from the identity then no matter the input kernel, the output kernel will bear some similarity to the input kernel.
Revised: On the other hand, if we control the deviation of the attention matrix product from the identity then no matter the input kernel, the output kernel will bear some similarity to the input kernel.

Original: So as long as the input kernel is non-degenerate and has full rank (regardless of duplicate tokens) so too will be the output kernel.
Revised: So as long as the input kernel is non-degenerate and has full rank (regardless of duplicate tokens) so too will be the output kernel.

Original: Recall that with attention matrices ( A l ) l , an input kernel matrix across T locations Σ 0 ∈ R T × T gets mapped, at depth l , to
Revised: Recall that with attention matrices ( A l ) l , an input kernel matrix across T locations Σ 0 ∈ R T × T gets mapped, at depth l , to

Original: [Equation]
Revised: [Equation]

Original: (11) where Π l = (cid:81) li =
Revised: (11) where Π l = (cid:81) li =

Original: 1 A i is the product of attention matrices.
Revised: 1 A i is the product of attention matrices.

Original: In SPA, we parameterise A l = L l L − 1 l − 1 for ( L l ) l corresponding to the Cholesky factors of some family of kernel matrices (Σ l ) l , with either uniform or exponentially decaying off diagonals:
Revised: In SPA, we parameterise A l = L l L − 1 l − 1 for ( L l ) l corresponding to the Cholesky factors of some family of kernel matrices (Σ l ) l , with either uniform or exponentially decaying off diagonals:

Original: U-SPA : Σ l ( ρ l )=(1 − ρ l ) I T + ρ l 11 ⊤ for 0 ≤ ρ ≤2.
Revised: U-SPA : Σ l ( ρ l )=(1 − ρ l ) I T + ρ l 11 ⊤ for 0 ≤ ρ ≤2.

Original: E-SPA : (cid:0) Σ l ( γ l ) (cid:1) i,j = exp ( − γ l | i − j | )) for γ ≥ 0 .
Revised: E-SPA : (cid:0) Σ l ( γ l ) (cid:1) i,j = exp ( − γ l | i − j | )) for γ ≥ 0 .

Original: It is important that A l = L l L − 1 l − 1 is constructed from two Cholesky matrices belonging to the same family, because Theorems 1 and 2 show in that case we will satisfy our non-negativity constraint on A l (which is computed through a softmax operation), and otherwise there is no prior reason to suppose that non-negativity will be satisfied.
Revised: It is important that A l = L l L − 1 l − 1 is constructed from two Cholesky matrices belonging to the same family, because Theorems 1 and 2 show in that case we will satisfy our non-negativity constraint on A l (which is computed through a softmax operation), and otherwise there is no prior reason to suppose that non-negativity will be satisfied.

Original: Therefore, in an ideal world, Σ 0 would be an element of our family of kernel matrices, so that we can set L 0 to be the Cholesky factor of Σ 0 .
Revised: Therefore, in an ideal world, Σ 0 would be an element of our family of kernel matrices, so that we can set L 0 to be the Cholesky factor of Σ 0 .

Original: Typically, Σ 0 = XX ⊤ /d 0 for input token embeddings
Revised: Typically, Σ 0 = XX ⊤ /d 0 for input token embeddings

Original: Repeated prob=0. Repeated prob=0.
Revised: Repeated prob=0. Repeated prob=0.

Original: Repeated prob=0. Repeated prob=0.
Revised: Repeated prob=0. Repeated prob=0.

Original: C o s i n e s i m il a r i t y
Revised: C o s i n e s i m il a r i t y

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e n
Revised: e n

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: e s
Revised: e s

Original: [Figure]
Revised: [Figure]

Original: X ∈ R T × d 0 , which are independently initialised for different tokens identites.
Revised: X ∈ R T × d 0 , which are independently initialised for different tokens identites.

Original: Thus, in the wide limit d → ∞ , (up to rescaling) Σ 0 will have diagonals equal to 1 , and off-diagonals equal to 1 if there is a repeated token and 0 else.
Revised: Thus, in the wide limit d → ∞ , (up to rescaling) Σ 0 will have diagonals equal to 1 , and off-diagonals equal to 1 if there is a repeated token and 0 else.

Original: We plot examples of such kernel matrices in Fig.
Revised: We plot examples of such kernel matrices in Fig.

Original: 5 for different fractions of repeated tokens r , under the assumption that repeated tokens occur independently of location.
Revised: 5 for different fractions of repeated tokens r , under the assumption that repeated tokens occur independently of location.

Original: Clearly, when r = 0 , we have Σ 

Revised: Clearly, when r = 0 , we have Σ

Original: = I is a member of both the uniform and exponential families of kernel matrices, corresponding to uniform off-diagonals of ρ 0 = 0 or exponentially decaying off-diagonals with rate γ 0 = ∞ .
Revised: = I is a member of both the uniform and exponential families of kernel matrices, corresponding to uniform off-diagonals of ρ 0 = 0 or exponentially decaying off-diagonals with rate γ 0 = ∞ .

Original: In this case we can set L 0 =
Revised: In this case we can set L 0 =

Original: I too.
Revised: I too.

Original: On the other hand, for r > 0 , it is in general difficult to say much more about an individual sequence’s Σ 0 , given that different sequences will have repeated tokens in different locations.
Revised: On the other hand, for r > 0 , it is in general difficult to say much more about an individual sequence’s Σ 0 , given that different sequences will have repeated tokens in different locations.

Original: Moreover, the naive approach of ignoring the repeated tokens and treating Σ 

Revised: Moreover, the naive approach of ignoring the repeated tokens and treating Σ

Original: = I leads to increasing diagonal values of Σ l (i.e. activation norms) at large depth without corrections, as shown in Fig.
Revised: = I leads to increasing diagonal values of Σ l (i.e. activation norms) at large depth without corrections, as shown in Fig.

Original: This imbalance between blocks could be problematic for training dynamics, and also is incompatible with frameworks like DKS and TAT which suppose that diagonal values of Σ l are constant across blocks and locations, usually set to 1.
Revised: This imbalance between blocks could be problematic for training dynamics, and also is incompatible with frameworks like DKS and TAT which suppose that diagonal values of Σ l are constant across blocks and locations, usually set to 1.

Original: To circumvent this, we will derive our modifications by considering the average input kernel matrix ¯ Σ 0 (averaged over different sequences), under the assumption that repeated tokens occur independent of location:
Revised: To circumvent this, we will derive our modifications by considering the average input kernel matrix ¯ Σ 0 (averaged over different sequences), under the assumption that repeated tokens occur independent of location:

Original: [Equation]
Revised: [Equation]

Original: By linearity of Eq.
Revised: By linearity of Eq.

Original: (11) in Σ 0 (and because we are controlling A l to be input-independent at initialisation in Section 3), it also follows that the average depth l kernel matrix ¯Σ l satisfies:
Revised: (11) in Σ 0 (and because we are controlling A l to be input-independent at initialisation in Section 3), it also follows that the average depth l kernel matrix ¯Σ l satisfies:

Original: [Equation]
Revised: [Equation]

Original: ) so we can modify our SPA approaches to control the average kernel matrix ¯Σ l instead.
Revised: ) so we can modify our SPA approaches to control the average kernel matrix ¯Σ l instead.

Original: For U-SPA, the situation is more straightforward as ¯Σ 0 is a uniform kernel matrix with ρ = r , so it suffices to simply let ρ 

Revised: For U-SPA, the situation is more straightforward as ¯Σ 0 is a uniform kernel matrix with ρ = r , so it suffices to simply let ρ

Original: = r instead of ρ 0 = 0 .
Revised: = r instead of ρ 0 = 0 .

Original: For E-SPA, we are unable to view ¯Σ 

Revised: For E-SPA, we are unable to view ¯Σ

Original: as having exponentially decaying off-diagonals, and so we keep γ 0 = ∞ which translates to L 

Revised: as having exponentially decaying off-diagonals, and so we keep γ 0 = ∞ which translates to L

Original: =
Revised: =

Original: I and Π l = L l .
Revised: I and Π l = L l .

Original: Instead, to help us understand the effect of repeated tokens (to motivate our modifications), we first expand on Eq.
Revised: Instead, to help us understand the effect of repeated tokens (to motivate our modifications), we first expand on Eq.

Original: (13) to simplify a little:
Revised: (13) to simplify a little:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where O T = 11 ⊤ − I T ∈ R T × T is 0 on the diagonal and 1 off the diagonal.
Revised: where O T = 11 ⊤ − I T ∈ R T × T is 0 on the diagonal and 1 off the diagonal.

Original: Note that all terms in Eq.
Revised: Note that all terms in Eq.

Original: (17) are easily computable (given that (cid:0) Σ l ( γ l ) (cid:1) i,j = exp ( − γ l | i − j | )) and
Revised: (17) are easily computable (given that (cid:0) Σ l ( γ l ) (cid:1) i,j = exp ( − γ l | i − j | )) and

Original: L l has an analytic form provided in Lemma 1).
Revised: L l has an analytic form provided in Lemma 1).

Original: So we can use Eq.
Revised: So we can use Eq.

Original: (17) to compute the expected diagonal (¯Σ l ) i,i for each location
Revised: (17) to compute the expected diagonal (¯Σ l ) i,i for each location

Original: i (where the expectation is taken over different sequences), which we denote by a diagonal matrix ¯ D l :
Revised: i (where the expectation is taken over different sequences), which we denote by a diagonal matrix ¯ D l :

Original: [Equation]
Revised: [Equation]

Original: Thus, we propose to replace A l =
Revised: Thus, we propose to replace A l =

Original: L l L − 1 l − 1 with A l = ¯ D − 12 l
Revised: L l L − 1 l − 1 with A l = ¯ D − 12 l

Original: L l L − 1 l − 1 ¯ D 12 l − 1 , setting ¯ D 

Revised: L l L − 1 l − 1 ¯ D 12 l − 1 , setting ¯ D

Original: = I by default.
Revised: = I by default.

Original: This means that our product Π l = (cid:81) li =
Revised: This means that our product Π l = (cid:81) li =

Original: 1 A i =
Revised: 1 A i =

Original: ¯ D − 12 l
Revised: ¯ D − 12 l

Original: L l , and Eq.
Revised: L l , and Eq.

Original: (13) is updated to:
Revised: (13) is updated to:

Original: [Equation]
Revised: [Equation]

Original: ) which has diagonals controlled to 1.
Revised: ) which has diagonals controlled to 1.

Original: Though our modifications for repeated tokens only consider averages across different sequences, we find that for individual sequences they still lead to well behaved diagonal values of Σ l , as shown in Fig.
Revised: Though our modifications for repeated tokens only consider averages across different sequences, we find that for individual sequences they still lead to well behaved diagonal values of Σ l , as shown in Fig.

Original: Moreover, we find that their effect on off-diagonals is still favourable for individual sequences, as shown in Fig.
Revised: Moreover, we find that their effect on off-diagonals is still favourable for individual sequences, as shown in Fig.

Original: C S ETTING ( ρ l ) l AND ( γ l ) l
Revised: C S ETTING ( ρ l ) l AND ( γ l ) l

Original: In this section we describe how we set the uniform off-diagonals ( ρ l ) l and the exponential decay rates ( γ l ) l in SPA, at different depths.
Revised: In this section we describe how we set the uniform off-diagonals ( ρ l ) l and the exponential decay rates ( γ l ) l in SPA, at different depths.

Original: Recall that Theorems 1 and 2 show that we are free to choose ( ρ l ) Ll =0 and ( γ l ) Ll =

Revised: Recall that Theorems 1 and 2 show that we are free to choose ( ρ l ) Ll =0 and ( γ l ) Ll =

Original: such that ( ρ l ) Ll = increase with depth and ( γ l ) Ll =0 decrease with depth.
Revised: such that ( ρ l ) Ll = increase with depth and ( γ l ) Ll =0 decrease with depth.

Original: Moreover, ρ 0 = 0 (or the shared token fraction r , as per Appendix B) and γ 0 = ∞ at the input layer, whilst we have found ρ L = 0 .
Revised: Moreover, ρ 0 = 0 (or the shared token fraction r , as per Appendix B) and γ 0 = ∞ at the input layer, whilst we have found ρ L = 0 .

Original: 8 and γ L = 0 .to be good default values for the last block.
Revised: 8 and γ L = 0 .to be good default values for the last block.

Original: For U-SPA, in terms of setting how ( ρ l ) l vary with depth, we tried different polynomial rates of increase with depth from ρ 0 to ρ L , but did not observe a noticeable difference across different rates so chose to increase from ρ 0 to ρ L linearly in depth.
Revised: For U-SPA, in terms of setting how ( ρ l ) l vary with depth, we tried different polynomial rates of increase with depth from ρ 0 to ρ L , but did not observe a noticeable difference across different rates so chose to increase from ρ 0 to ρ L linearly in depth.

Original: For E-SPA, we choose ( γ l ) l so that the diagonal elements of the attention matrices A l are constant across blocks, akin to using a constant shortcut weight α over different blocks.
Revised: For E-SPA, we choose ( γ l ) l so that the diagonal elements of the attention matrices A l are constant across blocks, akin to using a constant shortcut weight α over different blocks.

Original: From Theorem 3, we have that the diagonal entries of A l satisfy:
Revised: From Theorem 3, we have that the diagonal entries of A l satisfy:

Original: [Equation]
Revised: [Equation]

Original: where a ( γ )
Revised: where a ( γ )

Original: = (cid:112) 1 − exp ( − 2 γ ) with inverse γ ( a ) = − 12 log (1 − a 2 ) .
Revised: = (cid:112) 1 − exp ( − 2 γ ) with inverse γ ( a ) = − 12 log (1 − a 2 ) .

Original: This means that for a set of positive decreasing ( γ l ) l , there exist a corresponding set of decreasing ( a l ) l with values between 0 and 1 .
Revised: This means that for a set of positive decreasing ( γ l ) l , there exist a corresponding set of decreasing ( a l ) l with values between 0 and 1 .

Original: Because γ 0 = ∞ , we have a 0 = 1 , and likewise for a given γ L we can compute a L .
Revised: Because γ 0 = ∞ , we have a 0 = 1 , and likewise for a given γ L we can compute a L .

Original: Thus to have constant diagonal values of A l over different blocks l , we choose to set a l = a l
Revised: Thus to have constant diagonal values of A l over different blocks l , we choose to set a l = a l

Original: [Equation]
Revised: [Equation]

Original: for l ≤ L , and thus
Revised: for l ≤ L , and thus

Original: [Equation]
Revised: [Equation]

Original: We found this scheme to work well empirically, and note the similarity to other works which have discussed the choice of how to scale branches in residual architectures (Hayou et al., 2021).
Revised: We found this scheme to work well empirically, and note the similarity to other works which have discussed the choice of how to scale branches in residual architectures (Hayou et al., 2021).

Original: We leave further study of choosing how to set ( ρ l ) l and ( γ l ) l to future work.
Revised: We leave further study of choosing how to set ( ρ l ) l and ( γ l ) l to future work.

Original: D E-SPA ALGORITHM
Revised: D E-SPA ALGORITHM

Original: In Alg.
Revised: In Alg.

Original: 1 we present pseudocode to construct a trainable E-SPA masked attention layer.
Revised: 1 we present pseudocode to construct a trainable E-SPA masked attention layer.

Original: Algorithm 1: Modified E-SPA masked attention layer.
Revised: Algorithm 1: Modified E-SPA masked attention layer.

Original: Input: Input sequence representation X ∈ R T × d .
Revised: Input: Input sequence representation X ∈ R T × d .

Original: Output: Updated sequence representation Attn ( X ) ∈ R T × d .
Revised: Output: Updated sequence representation Attn ( X ) ∈ R T × d .

Original: Hyperparameters: Input γ in & output γ out exponential decay rates, γ in ≥ γ out .
Revised: Hyperparameters: Input γ in & output γ out exponential decay rates, γ in ≥ γ out .

Original: Number of heads h with head dimension d h = d/h .
Revised: Number of heads h with head dimension d h = d/h .

Original: Causal mask M ∈ R T × T s.t.
Revised: Causal mask M ∈ R T × T s.t.

Original: M i,j = 1 { i ≥ j } .
Revised: M i,j = 1 { i ≥ j } .

Original: Large constant Γ , e.g. 10 30 , to enforce causal mask.
Revised: Large constant Γ , e.g. 10 30 , to enforce causal mask.

Original: Trainable parameters: W Qn , W Kn , W Vn ∈ R d × d h , for 1 ≤ n ≤ h .
Revised: Trainable parameters: W Qn , W Kn , W Vn ∈ R d × d h , for 1 ≤ n ≤ h .

Original: W O ∈ R d × d .
Revised: W O ∈ R d × d .

Original: 1 Compute L in and L out to be the Cholesky factors of (Σ in i ) i,j = exp ( − γ in | i − j | ) and (Σ out i ) i,j = exp ( − γ out | i − j | ) respectively.
Revised: 1 Compute L in and L out to be the Cholesky factors of (Σ in i ) i,j = exp ( − γ in | i − j | ) and (Σ out i ) i,j = exp ( − γ out | i − j | ) respectively.

Original: 2 Set A =
Revised: 2 Set A =

Original: L out L − 1 in (using analytic form in Theorem 3) & decompose A = DP
Revised: L out L − 1 in (using analytic form in Theorem 3) & decompose A = DP

Original: where D diagonal & P has row sums 1.
Revised: where D diagonal & P has row sums 1.

Original: Denote B = log ( P ) .
Revised: Denote B = log ( P ) .

Original: 3 for head n ∈ { 1 , . . .
Revised: 3 for head n ∈ { 1 , . . .

Original: , h } do
Revised: , h } do

Original: Initialise W Kn , W Vn i.i.d.
Revised: Initialise W Kn , W Vn i.i.d.

Original: ∼ N (0 , 1 d ) (alternatively orthogonally).
Revised: ∼ N (0 , 1 d ) (alternatively orthogonally).

Original: Initialise W Qn
Revised: Initialise W Qn

Original: = 0 .
Revised: = 0 .

Original: Set Q n ( X ) =
Revised: Set Q n ( X ) =

Original: XW Qn , K n ( X ) =
Revised: XW Qn , K n ( X ) =

Original: XW Kn and V n ( X ) =
Revised: XW Kn and V n ( X ) =

Original: XW Vn .
Revised: XW Vn .

Original: Compute P n ( X ) = softmax (cid:16) M ◦ (cid:0) 1 √ d h Q n ( X ) K n ( X ) ⊤ + B (cid:1) − Γ(1 − M ) (cid:17)
Revised: Compute P n ( X ) = softmax (cid:16) M ◦ (cid:0) 1 √ d h Q n ( X ) K n ( X ) ⊤ + B (cid:1) − Γ(1 − M ) (cid:17)

Original: Set Attn n ( X ) =
Revised: Set Attn n ( X ) =

Original: DP n ( X ) · V n ( X ) 8 Initialise W O i.i.d.
Revised: DP n ( X ) · V n ( X ) 8 Initialise W O i.i.d.

Original: ∼ N (0 , 1 d ) (alternatively as an orthogonal matrix) 9 return Concat (cid:0) Attn 1 ( X ) , . . .
Revised: ∼ N (0 , 1 d ) (alternatively as an orthogonal matrix) 9 return Concat (cid:0) Attn 1 ( X ) , . . .

Original: , Attn h
Revised: , Attn h

Original: ( X ) (cid:1) W O
Revised: ( X ) (cid:1) W O

Original: E C OMPATIBILITY OF OUR APPROACHES WITH EXISTING POSITIONAL ENCODINGS
Revised: E C OMPATIBILITY OF OUR APPROACHES WITH EXISTING POSITIONAL ENCODINGS

Original: In Section 3.3, we showed how to control the attention matrix at initialisation, using bias matrices and location-dependent rescaling, as well as making the query-key dot product, 1 √ d k Q ( X ) K ( X ) ⊤ = 1 √ d k XW Q ( XW K ) ⊤ , zero at initialisation.
Revised: In Section 3.3, we showed how to control the attention matrix at initialisation, using bias matrices and location-dependent rescaling, as well as making the query-key dot product, 1 √ d k Q ( X ) K ( X ) ⊤ = 1 √ d k XW Q ( XW K ) ⊤ , zero at initialisation.

Original: This scheme is used in our SPA methods.
Revised: This scheme is used in our SPA methods.

Original: We detailed several ways to achieve this, and in practice we chose to initialise W Q to zero, and initialise W K as usual (Gaussian fan-in or orthogonal).
Revised: We detailed several ways to achieve this, and in practice we chose to initialise W Q to zero, and initialise W K as usual (Gaussian fan-in or orthogonal).

Original: In this section we show how zero initialising the query-key dot product is also possible when using two standard positional encodings: RoPE
Revised: In this section we show how zero initialising the query-key dot product is also possible when using two standard positional encodings: RoPE

Original: (Su et al., 2021) and Relative (Shaw et al., 2018; Huang et al., 2019; Dai et al., 2019).
Revised: (Su et al., 2021) and Relative (Shaw et al., 2018; Huang et al., 2019; Dai et al., 2019).

Original: This means that we can use our methods in Section 3.3 in combination with these positional encoders.
Revised: This means that we can use our methods in Section 3.3 in combination with these positional encoders.

Original: Let us denote the unscaled query-key dot product as S = XW Q ( XW K ) ⊤ .
Revised: Let us denote the unscaled query-key dot product as S = XW Q ( XW K ) ⊤ .

Original: For RoPE, the ( i, j ) query-key dot product, S i,j , is modified from
Revised: For RoPE, the ( i, j ) query-key dot product, S i,j , is modified from

Original: [Equation]
Revised: [Equation]

Original: to
Revised: to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where R i,j are some location-dependent rotation matrices defined in Su et al.
Revised: where R i,j are some location-dependent rotation matrices defined in Su et al.

Original: (2021), and X i denotes row
Revised: (2021), and X i denotes row

Original: i of the incoming representation X ∈ R T × d to the attention block.
Revised: i of the incoming representation X ∈ R T × d to the attention block.

Original: Clearly, Eq.
Revised: Clearly, Eq.

Original: (19) is 0 for all i, j if W Q is zero.
Revised: (19) is 0 for all i, j if W Q is zero.

Original: For Relative positional encoding, we take the scheme from (Dai et al., 2019).
Revised: For Relative positional encoding, we take the scheme from (Dai et al., 2019).

Original: In that case, the ( i, j ) query-key dot product, S i,j , is modified from
Revised: In that case, the ( i, j ) query-key dot product, S i,j , is modified from

Original: [Equation]
Revised: [Equation]

Original: to
Revised: to

Original: [Equation]
Revised: [Equation]

Original: where R i − j ∈ R d is fixed, and W K,R ∈ R d k × d , u , v ∈ R
Revised: where R i − j ∈ R d is fixed, and W K,R ∈ R d k × d , u , v ∈ R

Original: d k are trainable.
Revised: d k are trainable.

Original: Thus, we can achieve zero query-key dot products at initialisation if we initialise u , v = 0 , in addition to W Q = 0 .
Revised: Thus, we can achieve zero query-key dot products at initialisation if we initialise u , v = 0 , in addition to W Q = 0 .

Original: F U SING NORMALISED SKIP CONNECTIONS WITH E-SPA
Revised: F U SING NORMALISED SKIP CONNECTIONS WITH E-SPA

Original: In this section, we describe how to combine our E-SPA method with normalised skip connections in our attention blocks:
Revised: In this section, we describe how to combine our E-SPA method with normalised skip connections in our attention blocks:

Original: [Equation]
Revised: [Equation]

Original: (21) where α = 0 is the skipless setting that our methods are originally designed for.
Revised: (21) where α = 0 is the skipless setting that our methods are originally designed for.

Original: The general gist is that we will look at the combined effect, on the kernel matrix after the residual attention block, of the residual branch and the diagonal terms in the attention matrices for preserving signal propagation, and approximate the combination to match the setting where we are without skips, i.e. α = 0 .
Revised: The general gist is that we will look at the combined effect, on the kernel matrix after the residual attention block, of the residual branch and the diagonal terms in the attention matrices for preserving signal propagation, and approximate the combination to match the setting where we are without skips, i.e. α = 0 .

Original: To combine E-SPA with normalised skip connections, we consider how the cosine-similarity between two locations T 1 , T 2 ≤ T is affected through the normalised skip connection.
Revised: To combine E-SPA with normalised skip connections, we consider how the cosine-similarity between two locations T 1 , T 2 ≤ T is affected through the normalised skip connection.

Original: Suppose we have an input kernel matrix:
Revised: Suppose we have an input kernel matrix:

Original: [Equation]
Revised: [Equation]

Original: where (Σ) i,i = 1 , ∀ i and d is the width.
Revised: where (Σ) i,i = 1 , ∀ i and d is the width.

Original: Then after the residual attention block, Eq.
Revised: Then after the residual attention block, Eq.

Original: (21), we now have:
Revised: (21), we now have:

Original: [Equation]
Revised: [Equation]

Original: Either we have W V to be an orthogonal matrix sampled uniformly at random from the Haar measure, or W V i.i.d.
Revised: Either we have W V to be an orthogonal matrix sampled uniformly at random from the Haar measure, or W V i.i.d.

Original: ∼ N (0 , 1 d ) .
Revised: ∼ N (0 , 1 d ) .

Original: In both cases we have 1 d AXW V W V
Revised: In both cases we have 1 d AXW V W V

Original: ⊤ X ⊤
Revised: ⊤ X ⊤

Original: A ⊤ going towards 1 d AXX ⊤ A ⊤ , and the cross terms Eq.
Revised: A ⊤ going towards 1 d AXX ⊤ A ⊤ , and the cross terms Eq.

Original: (23) converging to 0 for large d .
Revised: (23) converging to 0 for large d .

Original: Thus, we can consider the large d approximation:
Revised: Thus, we can consider the large d approximation:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, if we look at the inner product between the T 1 and T 2 locations for locations T 1 ̸ = T 2 such that T 1 , T 2 > 1 , we have:
Revised: Then, if we look at the inner product between the T 1 and T 2 locations for locations T 1 ̸ = T 2 such that T 1 , T 2 > 1 , we have:

Original: [Equation]
Revised: [Equation]

Original: where λ =
Revised: where λ =

Original: A T 1 ,T 1 =
Revised: A T 1 ,T 1 =

Original: A T 2 ,T 2 is the constant diagonal of the attention matrices in E-SPA, c.f.
Revised: A T 2 ,T 2 is the constant diagonal of the attention matrices in E-SPA, c.f.

Original: Theorem 3, and ν = α 2 + (1 − α 2 ) λ 2 .
Revised: Theorem 3, and ν = α 2 + (1 − α 2 ) λ 2 .

Original: Moreover, we have defined δ = (cid:80) i ̸
Revised: Moreover, we have defined δ = (cid:80) i ̸

Original: = T 1 ∪ j ̸ = T 2 A T 1 ,i Σ i,j A T 2 ,j as it is a term that is not possible to control with only knowledge of Σ T 1 ,T 2 and will vary from sequence to sequence, and hence we argue can be discarded from a signal propagation perspective.
Revised: = T 1 ∪ j ̸ = T 2 A T 1 ,i Σ i,j A T 2 ,j as it is a term that is not possible to control with only knowledge of Σ T 1 ,T 2 and will vary from sequence to sequence, and hence we argue can be discarded from a signal propagation perspective.

Original: For example, one could consider a sequence where the kernel matrix Σ has all off-diagonals equal to 0 apart from Σ T 1 ,T 2 , and T 1 could be sufficiently distant from T 2 such that there is no i for which A T 1 ,i and A T 2 ,i are large at the same time (which can be seen using the analytic form of A given in Theorem 3).
Revised: For example, one could consider a sequence where the kernel matrix Σ has all off-diagonals equal to 0 apart from Σ T 1 ,T 2 , and T 1 could be sufficiently distant from T 2 such that there is no i for which A T 1 ,i and A T 2 ,i are large at the same time (which can be seen using the analytic form of A given in Theorem 3).

Original: Thus, from Eq.
Revised: Thus, from Eq.

Original: (30), we see that after an residual attention block, an input cosine similarity of Σ T 1 ,Tbetween locations T 1 , T 2 is diluted by a factor of
Revised: (30), we see that after an residual attention block, an input cosine similarity of Σ T 1 ,Tbetween locations T 1 , T 2 is diluted by a factor of

Original: [Equation]
Revised: [Equation]

Original: , with shortcut weight α and attention diagonal probability λ .
Revised: , with shortcut weight α and attention diagonal probability λ .

Original: Our approximation then seeks to preserve this factor ν when α > 0 to match the skipless case.
Revised: Our approximation then seeks to preserve this factor ν when α > 0 to match the skipless case.

Original: Now, in the skipless case α = 0 described in Section 3, at block l we suppose that the incoming kernel matrix Σ has exponentially decaying off-diagonals with rate γ l − 1 , and that we construct the attention matrix A so that the output kernel matrix has exponentially decaying off diagonals with rate γ l .
Revised: Now, in the skipless case α = 0 described in Section 3, at block l we suppose that the incoming kernel matrix Σ has exponentially decaying off-diagonals with rate γ l − 1 , and that we construct the attention matrix A so that the output kernel matrix has exponentially decaying off diagonals with rate γ l .

Original: From Theorem 3, we see that this gives diagonal entries of A to be:
Revised: From Theorem 3, we see that this gives diagonal entries of A to be:

Original: [Equation]
Revised: [Equation]

Original: where a ( γ )
Revised: where a ( γ )

Original: = (cid:112) 1 − exp ( − 2 γ ) .
Revised: = (cid:112) 1 − exp ( − 2 γ ) .

Original: Thus, to preserve ν to match the case for α = 0 , if we have shortcut weight α > 0 we need the attention matrix diagonal probability λ α to satisfy:
Revised: Thus, to preserve ν to match the case for α = 0 , if we have shortcut weight α > 0 we need the attention matrix diagonal probability λ α to satisfy:

Original: [Equation]
Revised: [Equation]

Original: In turn, this means that we need to choose γ l,α , such that:
Revised: In turn, this means that we need to choose γ l,α , such that:

Original: [Equation]
Revised: [Equation]

Original: Inverting the definition of a ( γ ) , we see we need to set γ l,α :
Revised: Inverting the definition of a ( γ ) , we see we need to set γ l,α :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: To summarise, if we have a sequence ( γ l ) l of decreasing exponential decay rates, our proposed approximation when using shortcut weights α at block l is, using the notation of Alg. 1, to set γ in = γ l − 1 as normal, and to set γ out = γ l,α from Eq.
Revised: To summarise, if we have a sequence ( γ l ) l of decreasing exponential decay rates, our proposed approximation when using shortcut weights α at block l is, using the notation of Alg. 1, to set γ in = γ l − 1 as normal, and to set γ out = γ l,α from Eq.

Original: (33) in order to preserve the signal propagation from the combined residual attention block Eq.
Revised: (33) in order to preserve the signal propagation from the combined residual attention block Eq.

Original: We see that this approximation reduces to the standard skipless setting when α = 0 , as a sanity check.
Revised: We see that this approximation reduces to the standard skipless setting when α = 0 , as a sanity check.

Original: G A DDITIONAL RESULTS
Revised: G A DDITIONAL RESULTS

Original: In this section we present additional results and ablations that were not included in Section 4.
Revised: In this section we present additional results and ablations that were not included in Section 4.

Original: Normalised kernel matrix evolution for attention-only transformers with skips and normalisation In Fig.
Revised: Normalised kernel matrix evolution for attention-only transformers with skips and normalisation In Fig.

Original: 7 we plot the evolution of normalised kernel matrices for transformers with skips and or RMSNorm normalisation, in addition to those for vanilla transformers in Fig.
Revised: 7 we plot the evolution of normalised kernel matrices for transformers with skips and or RMSNorm normalisation, in addition to those for vanilla transformers in Fig.

Original: We see that both skipless with RMSNorm (fourth row) and Post-LN (bottom row) converge to rank collapse at larger depths.
Revised: We see that both skipless with RMSNorm (fourth row) and Post-LN (bottom row) converge to rank collapse at larger depths.

Original: The degeneration of skipless with RMSNorm is expected from the results of (Dong et al., 2021), and while the convergence to rank collapse is slower for Post-LN, it is still expected (Hayou et al., 2021;
Revised: The degeneration of skipless with RMSNorm is expected from the results of (Dong et al., 2021), and while the convergence to rank collapse is slower for Post-LN, it is still expected (Hayou et al., 2021;

Original: Noci et al., 2022).
Revised: Noci et al., 2022).

Original: This is because the residual and shortcut branches are effectively given a constant weighting at all blocks in Post-LN, even as the network’s depth increases.
Revised: This is because the residual and shortcut branches are effectively given a constant weighting at all blocks in Post-LN, even as the network’s depth increases.

Original: On the other hand, Pre-LN observes sensible signal propagation even at depth 100, as the positioning of the LN in the residual branch effectively downweights the residual branch at later blocks (De & Smith, 2020).
Revised: On the other hand, Pre-LN observes sensible signal propagation even at depth 100, as the positioning of the LN in the residual branch effectively downweights the residual branch at later blocks (De & Smith, 2020).

Original: Likewise, Pre-LN with skip weight α = 0 .
Revised: Likewise, Pre-LN with skip weight α = 0 .

Original: 98 also observes faithful signal propagation, because each block is effectively downweighted.
Revised: 98 also observes faithful signal propagation, because each block is effectively downweighted.

Original: This effect means that standard Pre-LN’s
Revised: This effect means that standard Pre-LN’s

Original: (fifth row) kernel matrix increases elementwise faster with depth than Pre-LN with normalised skips (sixth row), despite both kernel matrices being qualitatively similar at block 100.
Revised: (fifth row) kernel matrix increases elementwise faster with depth than Pre-LN with normalised skips (sixth row), despite both kernel matrices being qualitatively similar at block 100.

Original: Note, both PreLN kernel matrices also have a positional encoding like recency bias , like our main method, E-SPA (third row).
Revised: Note, both PreLN kernel matrices also have a positional encoding like recency bias , like our main method, E-SPA (third row).

Original: Validation perplexity across different skipless methods In Table 1 we compared the training speeds for our skipless methods, and found that E-SPA outperforms both U-SPA and Value-Skipinit.
Revised: Validation perplexity across different skipless methods In Table 1 we compared the training speeds for our skipless methods, and found that E-SPA outperforms both U-SPA and Value-Skipinit.

Original: In particular, we found E-SPA with a DKS-transformed GeLU activation without LN to perform best.
Revised: In particular, we found E-SPA with a DKS-transformed GeLU activation without LN to perform best.

Original: In Table 4, we present the corresponding results but for validation perplexity.
Revised: In Table 4, we present the corresponding results but for validation perplexity.

Original: Again, we see that E-SPA is the best performing of our attention modifications, but in this case TAT with Leaky
Revised: Again, we see that E-SPA is the best performing of our attention modifications, but in this case TAT with Leaky

Original: ReLU and no LN matches or outperforms DKS with GeLU.
Revised: ReLU and no LN matches or outperforms DKS with GeLU.

Original: Sensitivity to query-key initialisation scale Recall in Section 3.3 that for attention layers using
Revised: Sensitivity to query-key initialisation scale Recall in Section 3.3 that for attention layers using

Original: SPA we seek to initialise weights such that the query-key dot product, 1 √ d k XW Q ( XW K ) ⊤ , is zero or small at initialisation.
Revised: SPA we seek to initialise weights such that the query-key dot product, 1 √ d k XW Q ( XW K ) ⊤ , is zero or small at initialisation.

Original: In our main experiments, we achieve this by initialising W Q = 0 , and letting W K to be initialised as normal.
Revised: In our main experiments, we achieve this by initialising W Q = 0 , and letting W K to be initialised as normal.

Original: In Fig.
Revised: In Fig.

Original: 8, we assess the sensitivity of our E-SPA scheme to the scale of non-zero attention dot product, when W K , W Q are both orthogonally initialised but with scale σ (i.e. at each layer, both W K , W Q are initialised as two independent uniform orthgonal matrices multiplied by σ ).
Revised: 8, we assess the sensitivity of our E-SPA scheme to the scale of non-zero attention dot product, when W K , W Q are both orthogonally initialised but with scale σ (i.e. at each layer, both W K , W Q are initialised as two independent uniform orthgonal matrices multiplied by σ ).

Original: We see that for small initialisation scales, there is little effect
Revised: We see that for small initialisation scales, there is little effect

Original: [Figure]
Revised: [Figure]

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: [Figure]
Revised: [Figure]

Original: n i f o r m S P A
Revised: n i f o r m S P A

Original: [Figure]
Revised: [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: n e n t i a l S
Revised: n e n t i a l S

Original: P A
Revised: P A

Original: [Figure]
Revised: [Figure]

Original: S k i p
Revised: S k i p

Original: l e ss + L N
Revised: l e ss + L N

Original: [Figure]
Revised: [Figure]

Original: r e - L N
Revised: r e - L N

Original: [Figure]
Revised: [Figure]

Original: P r
Revised: P r

Original: [Figure]
Revised: [Figure]

Original: - L N w i
Revised: - L N w i

Original: t h s k
Revised: t h s k

Original: i p w e i g h t
Revised: i p w e i g h t

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: i n
Revised: i n

Original: [Figure]
Revised: [Figure]

Original: s i m il a r i t y of varying initial scale, but that training performance degrades at larger scales, when our attention matrix reverse engineering in Section 3.3 (which expects small or zero query-key dot product at initialisation) is less precise.
Revised: s i m il a r i t y of varying initial scale, but that training performance degrades at larger scales, when our attention matrix reverse engineering in Section 3.3 (which expects small or zero query-key dot product at initialisation) is less precise.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: n l o ss
Revised: n l o ss

Original: [Figure]
Revised: [Figure]

Original: Ablation over orthogonal initialisation Recall in Section 3 that our kernel matrix evolution for attention-only transformers is exact at finite widths using orthogonally initialised weight matrices, and will be approximate using standard (Gaussian) fan-in initialised In Fig.
Revised: Ablation over orthogonal initialisation Recall in Section 3 that our kernel matrix evolution for attention-only transformers is exact at finite widths using orthogonally initialised weight matrices, and will be approximate using standard (Gaussian) fan-in initialised In Fig.

Original: 9, we ablate over using orthogonally initialised weight matrices, compared to Gaussian fan-in initialisation.
Revised: 9, we ablate over using orthogonally initialised weight matrices, compared to Gaussian fan-in initialisation.

Original: Across activations, we see that E-SPA with orthogonal initialisation slightly outperforms Gaussian fan-in initialisation (by around 0.15 train loss).
Revised: Across activations, we see that E-SPA with orthogonal initialisation slightly outperforms Gaussian fan-in initialisation (by around 0.15 train loss).

Original: [Figure]
Revised: [Figure]

Original: Training step
Revised: Training step

Original: [Figure]
Revised: [Figure]

Original: W
Revised: W

Original: T
Revised: T

Original: T r a i n l o ss
Revised: T r a i n l o ss

Original: DKS-GeLU,orth=
Revised: DKS-GeLU,orth=

Original: False
Revised: False

Original: DKS-GeLU,orth=
Revised: DKS-GeLU,orth=

Original: True
Revised: True

Original: TAT-LReLU,orth=
Revised: TAT-LReLU,orth=

Original: False
Revised: False

Original: TAT-LReLU,orth=
Revised: TAT-LReLU,orth=

Original: True
Revised: True

Original: GeLU,orth=
Revised: GeLU,orth=

Original: False
Revised: False

Original: GeLU,orth=
Revised: GeLU,orth=

Original: True
Revised: True

Original: Ablation over positional encodings As noted in Appendix E, our methods are compatible with several standard positional encodings, and by default all our experiments use the popular RoPE
Revised: Ablation over positional encodings As noted in Appendix E, our methods are compatible with several standard positional encodings, and by default all our experiments use the popular RoPE

Original: (Su et al., 2021) positional encoding.
Revised: (Su et al., 2021) positional encoding.

Original: In Fig.
Revised: In Fig.

Original: 10 and Table 5, we assess the effect of removing positional encodings, on training and validation performance respectively, in our skipless methods.
Revised: 10 and Table 5, we assess the effect of removing positional encodings, on training and validation performance respectively, in our skipless methods.

Original: We see that all methods are improved when combined with RoPE, however the improvement is most mild in E-SPA, which as discussed has an in-built recency bias akin to a positional encoder.
Revised: We see that all methods are improved when combined with RoPE, however the improvement is most mild in E-SPA, which as discussed has an in-built recency bias akin to a positional encoder.

Original: Moreover, ESPA without additional positional encoding still outperforms all other approaches, including U-SPA (which on its own has no notion of position) with RoPE.
Revised: Moreover, ESPA without additional positional encoding still outperforms all other approaches, including U-SPA (which on its own has no notion of position) with RoPE.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Training step
Revised: Training step

Original: [Table]
Revised: [Table]

Original: W
Revised: W

Original: T
Revised: T

Original: T r a i n l o ss
Revised: T r a i n l o ss

Original: [Table]
Revised: [Table]

Original: Stable residual connection (Hayou et al., 2021;
Revised: Stable residual connection (Hayou et al., 2021;

Original: Noci et al., 2022) on WikiText-

Revised: Noci et al., 2022) on WikiText-

Original: In Fig.
Revised: In Fig.

Original: 11, we provide an equivalent plot to Fig.
Revised: 11, we provide an equivalent plot to Fig.

Original: 3 using a Stable ResNet (Hayou et al., 2021) rescaling of the shortcut weights ( α = 1 , β
Revised: 3 using a Stable ResNet (Hayou et al., 2021) rescaling of the shortcut weights ( α = 1 , β

Original: = O ( 1 √ L ) ).
Revised: = O ( 1 √ L ) ).

Original: In this case, the shortcut weight is always 1 and the residual weight β is uniform across blocks and scales as O ( 1 √ L ) in depth L .
Revised: In this case, the shortcut weight is always 1 and the residual weight β is uniform across blocks and scales as O ( 1 √ L ) in depth L .

Original: Hayou et al.
Revised: Hayou et al.

Original: (2021) showed that such a scaling leads to non-degenerate signal propagation in large depth MLPs/CNNs without normalisation, and Noci et al.
Revised: (2021) showed that such a scaling leads to non-degenerate signal propagation in large depth MLPs/CNNs without normalisation, and Noci et al.

Original: (2022) showed that the stable scaling prevents rank collapse in transformers without normalisation.
Revised: (2022) showed that the stable scaling prevents rank collapse in transformers without normalisation.

Original: We see in Fig.
Revised: We see in Fig.

Original: 11 that for large enough β , the stable residual weighting with normalisation matches the training speed of the default transformer (which is unsurprising given that once β
Revised: 11 that for large enough β , the stable residual weighting with normalisation matches the training speed of the default transformer (which is unsurprising given that once β

Original: = 1 it is exactly default Pre-LN).
Revised: = 1 it is exactly default Pre-LN).

Original: However, there is a small but consistent gap without normalisation (with optimal β = 0 . 1 ).
Revised: However, there is a small but consistent gap without normalisation (with optimal β = 0 . 1 ).

Original: Here, L = 36 , so 1 √ L = 16 ≈ 0 .
Revised: Here, L = 36 , so 1 √ L = 16 ≈ 0 .

Original: 17 , or alternatively if we count the 72 nonlinear layers (one self-attention and element-wise nonlinearity for each transformer block), we have 1 √ 

Revised: 17 , or alternatively if we count the 72 nonlinear layers (one self-attention and element-wise nonlinearity for each transformer block), we have 1 √

Original: ≈ 0 .
Revised: ≈ 0 .

Original: Residual weight 1.
Revised: Residual weight 1.

Original: W T
Revised: W T

Original: T r a i n l o ss
Revised: T r a i n l o ss

Original: Stable Resnet, =
Revised: Stable Resnet, =

Original: ln=
Revised: ln=

Original: none
Revised: none

Original: ln=rmsnorm default (100k)
Revised: ln=rmsnorm default (100k)

Original: Different shortcut weights for MLP and self-attention blocks Recall that in a transformer block
Revised: Different shortcut weights for MLP and self-attention blocks Recall that in a transformer block

Original: Eq.
Revised: Eq.

Original: (1), there are two distinct skip connections: one for the self-attention block and one for the MLP block.
Revised: (1), there are two distinct skip connections: one for the self-attention block and one for the MLP block.

Original: Moreover, we observed a training speed gap when we remove both skip connections in Fig.
Revised: Moreover, we observed a training speed gap when we remove both skip connections in Fig.

Original: This leads us to ask if it is possible that only one of the skips, MLP or attention, is causing this gap in training speed.
Revised: This leads us to ask if it is possible that only one of the skips, MLP or attention, is causing this gap in training speed.

Original: In Fig.
Revised: In Fig.

Original: 12, we investigate this by varying the MLP shortcut weight for skipless attention blocks (left) and varying the attention shortcut weight for skipless MLP blocks (right).
Revised: 12, we investigate this by varying the MLP shortcut weight for skipless attention blocks (left) and varying the attention shortcut weight for skipless MLP blocks (right).

Original: For all skip connections (for both MLP and self-attention blocks), we use a normalised skip connection ( β = √ 1 − α 2 ) .
Revised: For all skip connections (for both MLP and self-attention blocks), we use a normalised skip connection ( β = √ 1 − α 2 ) .

Original: We observe that removing either skip connection results in a comparable loss of training speed (the default Pre-LN on WikiText-103 obtained train loss of 1.76 after 100K steps), although having one is still better than having neither.
Revised: We observe that removing either skip connection results in a comparable loss of training speed (the default Pre-LN on WikiText-103 obtained train loss of 1.76 after 100K steps), although having one is still better than having neither.

Original: Moreover, we observe that the attention and
Revised: Moreover, we observe that the attention and

Original: MLP blocks prefer slightly different shortcut weights, whereas our experiments in Figs.
Revised: MLP blocks prefer slightly different shortcut weights, whereas our experiments in Figs.

Original: 3 anduse a joint weighting for both.
Revised: 3 anduse a joint weighting for both.

Original: [Figure]
Revised: [Figure]

Original: Training performance on C4 In Fig.
Revised: Training performance on C4 In Fig.

Original: 13 we compare the training performance of our various vanilla transfomers to the default Pre-LN transformer on C4.
Revised: 13 we compare the training performance of our various vanilla transfomers to the default Pre-LN transformer on C4.

Original: This is akin to Fig. 4a, which compared validation performance.
Revised: This is akin to Fig. 4a, which compared validation performance.

Original: 0K 10K 20K 30K 40K 50K
Revised: [Figure]

Original: Training step 3.
Revised: Training step 3.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: C
Revised: C

Original: T r a i n l o ss skip+LN e-spa u-spa value skipinit
Revised: T r a i n l o ss skip+LN e-spa u-spa value skipinit

Original: Downstream tasks Typically, transformers are pre-trained on a large corpus of data before evaluation on a set of downstream tasks.
Revised: Downstream tasks Typically, transformers are pre-trained on a large corpus of data before evaluation on a set of downstream tasks.

Original: To assess whether our conclusions about training performance in pre-training transfer over to downstream tasks, we assess models trained on C4 on 5 common sense downstream tasks:
Revised: To assess whether our conclusions about training performance in pre-training transfer over to downstream tasks, we assess models trained on C4 on 5 common sense downstream tasks:

Original: BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2020), PIQA (Bisk et al., 2020), and SIQA (Sap et al., 2019).
Revised: BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2020), PIQA (Bisk et al., 2020), and SIQA (Sap et al., 2019).

Original: These datasets are commonly used to evaluate large pre-trained transformers (Brown et al., 2020;
Revised: These datasets are commonly used to evaluate large pre-trained transformers (Brown et al., 2020;

Original: Rae et al., 2021;
Revised: Rae et al., 2021;

Original: Smith et al., 2022; Hoffmann et al., 2022).
Revised: Smith et al., 2022; Hoffmann et al., 2022).

Original: In Table 6, we see that the conclusions of pre-training on C4 largely carry over to the downstream tasks:
Revised: In Table 6, we see that the conclusions of pre-training on C4 largely carry over to the downstream tasks:

Original: • Among transformers trained for the same number of steps (50k), E-SPA beats U-SPA and Value-SkipInit on 4 out of 5 downstream tasks.
Revised: • Among transformers trained for the same number of steps (50k), E-SPA beats U-SPA and Value-SkipInit on 4 out of 5 downstream tasks.

Original: However, the default transformer outperforms skipless transformers on all tasks with the same amount of training.
Revised: However, the default transformer outperforms skipless transformers on all tasks with the same amount of training.

Original: • With around 5 times longer training (200K and 300K steps), E-SPA achieves similar performance on downstream tasks to the standard transformer, outperforming on 2 out oftasks (Winogrande and PIQA).
Revised: • With around 5 times longer training (200K and 300K steps), E-SPA achieves similar performance on downstream tasks to the standard transformer, outperforming on 2 out oftasks (Winogrande and PIQA).

Original: [Table]
Revised: [Table]

Original: Longer training on WikiText-

Revised: Longer training on WikiText-

Original: In Fig.
Revised: In Fig.

Original: 14 we see that 4.5x more training allows a vanilla ESPA transformer to match the validation performance of a standard transformer on WikiText-103.
Revised: 14 we see that 4.5x more training allows a vanilla ESPA transformer to match the validation performance of a standard transformer on WikiText-103.

Original: This mirrors our findings on C4 in Fig. 4b.
Revised: This mirrors our findings on C4 in Fig. 4b.

Original: [Figure]
Revised: Kernel matrix evolution during training Fig. 15 shows the evolution of the empiricallycomputed normalized kernel matrix during training of a (finite width) vanilla E-SPA transformer 0K 100K 200K 300K 400K 500K 600K

Original: Training step 1.
Revised: Training step 1.

Original: [Figure]

Original: W T
Revised: W T

Original: T r a i n l o ss 100k steps 200k steps 400k steps 600k steps default (100k) 
Revised: T r a i n l o ss 100k steps 200k steps 400k steps 600k steps default (100k) on WikiText-103.

Revised: We note that the untrained network shows good agreement with Fig.

Revised: 1, despite the fact

Revised: that Fig. 1 is computed for an attention-only network in the infinite width limit.

Revised: We also note that while significant changes to the kernel matrix occur during training, it retains the property of being larger close the diagonal

Revised: [Figure]

Original: H I MPLEMENTATION DETAILS
Revised: H I MPLEMENTATION DETAILS

Original: We first describe all additional general implementation details for our experiments, before going into details relevant for individual results.
Revised: We first describe all additional general implementation details for our experiments, before going into details relevant for individual results.

Original: H. G ENERAL IMPLEMENTATION DETAILS
Revised: H. G ENERAL IMPLEMENTATION DETAILS

Original: Datasets We present experiments on WikiText-103 (Merity et al., 2017) and C4 (Raffel et al., 2019).
Revised: Datasets We present experiments on WikiText-103 (Merity et al., 2017) and C4 (Raffel et al., 2019).

Original: For both datasets, we use the SentencePiece tokeniser (Kudo & Richardson, 2018) with vocabulary size | V | = 32 , 000 .
Revised: For both datasets, we use the SentencePiece tokeniser (Kudo & Richardson, 2018) with vocabulary size | V | = 32 , 000 .

Original: For WikiText-103 we use sequence length of 512 for both training and validation.
Revised: For WikiText-103 we use sequence length of 512 for both training and validation.

Original: For C4, the sequence length is 2048 for both.
Revised: For C4, the sequence length is 2048 for both.

Original: Embedding By default the embedding layer E ∈ R | V |× d 0 is fan-out initialised at initialisation, so that E i,j i.i.d.
Revised: Embedding By default the embedding layer E ∈ R | V |× d 0 is fan-out initialised at initialisation, so that E i,j i.i.d.

Original: ∼ N (0 , 1 d 0 ) .
Revised: ∼ N (0 , 1 d 0 ) .

Original: This means that each embedding entry has scale 1 d 0 at initialisation, whereas our Gram matrices (which are dimension normalised) expect each entry to have scale 1 at initialisation.
Revised: This means that each embedding entry has scale 1 d 0 at initialisation, whereas our Gram matrices (which are dimension normalised) expect each entry to have scale 1 at initialisation.

Original: To get by this, in our proposed vanilla methods we rescale the embedding matrix by a fixed factor of √ d 0 , so that E ← √ d 0 E which is then treated as a lookup table.
Revised: To get by this, in our proposed vanilla methods we rescale the embedding matrix by a fixed factor of √ d 0 , so that E ← √ d 0 E which is then treated as a lookup table.

Original: At initialisation, this has the same effect using RMSNorm on the embedding layer.
Revised: At initialisation, this has the same effect using RMSNorm on the embedding layer.

Original: In all our experiments, the unembedding layer weights are shared with the embedding weights.
Revised: In all our experiments, the unembedding layer weights are shared with the embedding weights.

Original: We do not use the embedding layer positional encoding introduced by Vaswani et al.
Revised: We do not use the embedding layer positional encoding introduced by Vaswani et al.

Original: Model
Revised: Model

Original: In all our experiments apart from Table 3, the model width d = 1024 across all blocks.
Revised: In all our experiments apart from Table 3, the model width d = 1024 across all blocks.

Original: We use 8 head multi-head attention, so that d k = 128 .
Revised: We use 8 head multi-head attention, so that d k = 128 .

Original: The MLP block consists of a single hidden layer with width 4 d , with input and output dimensions both equal to d .
Revised: The MLP block consists of a single hidden layer with width 4 d , with input and output dimensions both equal to d .

Original: All our experiments use the PreLN transformer block Eq.
Revised: All our experiments use the PreLN transformer block Eq.

Original: On WikiText-103, we use a 36 block transformer by default for all experiments apart from Table 3.
Revised: On WikiText-103, we use a 36 block transformer by default for all experiments apart from Table 3.

Original: On C4, we use a 32 block transformer due to memory constraints.
Revised: On C4, we use a 32 block transformer due to memory constraints.

Original: Any normalisation layer we consider is RMSNorm (Zhang & Sennrich, 2019), which is simpler than Layer Normalisation (Ba et al., 2016) and is commonly used in transformers (Rae et al., 2021).
Revised: Any normalisation layer we consider is RMSNorm (Zhang & Sennrich, 2019), which is simpler than Layer Normalisation (Ba et al., 2016) and is commonly used in transformers (Rae et al., 2021).

Original: By default, our models use RoPE positional encoder (Su et al., 2021)
Revised: By default, our models use RoPE positional encoder (Su et al., 2021)

Original: (apart from the ablation in
Revised: (apart from the ablation in

Original: Fig. 10).
Revised: Fig. 10).

Original: Parameter initialisations By default, all weight matrices are fan-in initialised N (0 , σ 2 fan-in ) with σ = 1 .
Revised: Parameter initialisations By default, all weight matrices are fan-in initialised N (0 , σ 2 fan-in ) with σ = 1 .

Original: The two exceptions for this are: 1) when using orthogonal intiailisation, we use the scaledcorrected uniform orthogonal initialisation (Martens et al., 2021) with scale σ = 1 (which for square matrices is just an orthogonal matrix sampled from the Haar measure Meckes (2019) multiplied by σ ), and 2) for the parameter matrix immediately after the activation we set σ to take input activation norm (“q-values” or diagonals of Gram matrices) of 1 to output activation norm of 1. {+
+}
Revised: The two exceptions for this are: 1) when using orthogonal intiailisation, we use the scaledcorrected uniform orthogonal initialisation (Martens et al., 2021) with scale σ = 1 (which for square matrices is just an orthogonal matrix sampled from the Haar measure Meckes (2019) multiplied by σ ), and 2) for the parameter matrix immediately after the activation we set σ to take input activation norm (“q-values” or diagonals of Gram matrices) of 1 to output activation norm of 1. {+
+}

Original: All bias parameters are initialised to 0.
Revised: All bias parameters are initialised to 0.

Original: (Transformed) Activations For DKS (Martens et al., 2021)
Revised: (Transformed) Activations For DKS (Martens et al., 2021)

Original: we set slope parameter ξ = 5 , and for TAT (Zhang et al., 2022) with leaky ReLU, we set η = 0 .
Revised: we set slope parameter ξ = 5 , and for TAT (Zhang et al., 2022) with leaky ReLU, we set η = 0 .

Original: Both values were chosen by a small hyperparameter sweep on WikiText-103.
Revised: Both values were chosen by a small hyperparameter sweep on WikiText-103.

Original: The DKS and TAT transformations are chosen without consideration of the attention blocks, where the transformer can be viewed as an MLP (potentially with residual connections).
Revised: The DKS and TAT transformations are chosen without consideration of the attention blocks, where the transformer can be viewed as an MLP (potentially with residual connections).

Original: Unless stated otherwise, all skipless transformers used a
Revised: Unless stated otherwise, all skipless transformers used a

Original: DKS-transformed GeLU as the nonlinearity in the MLP block by default.
Revised: DKS-transformed GeLU as the nonlinearity in the MLP block by default.

Original: Loss We perform next-token prediction, with softmax cross entropy loss.
Revised: Loss We perform next-token prediction, with softmax cross entropy loss.

Original: More concretely, if
Revised: More concretely, if

Original: X L ∈ R T ×
Revised: X L ∈ R T ×

Original: d denotes our final layer representation and E ∈ R | V |× d denotes our embedding layer, then we obtain logits X L E ⊤ ∈ R T ×| V | for each location for a | V | -way classification.
Revised: d denotes our final layer representation and E ∈ R | V |× d denotes our embedding layer, then we obtain logits X L E ⊤ ∈ R T ×| V | for each location for a | V | -way classification.

Original: The loss is softmax cross entropy, obtained by using each location i to predict the identity, in | V | , of the token at location i + 1 .
Revised: The loss is softmax cross entropy, obtained by using each location i to predict the identity, in | V | , of the token at location i + 1 .

Original: In our training loss results, we use an exponential moving average with smoothing factor 0.01, to reduce randomness from mini-batching.
Revised: In our training loss results, we use an exponential moving average with smoothing factor 0.01, to reduce randomness from mini-batching.

Original: Optimiser We use Adam optimiser (Kingma & Ba, 2014) with global gradient clipping of 0.1 by default (Pascanu et al., 2013).
Revised: Optimiser We use Adam optimiser (Kingma & Ba, 2014) with global gradient clipping of 0.1 by default (Pascanu et al., 2013).

Original: We do not use weight decay in our experiments.
Revised: We do not use weight decay in our experiments.

Original: Training We use mini-batching of 16 sequences for WikiText-103 and 8 for C4, due to memory constraints.
Revised: Training We use mini-batching of 16 sequences for WikiText-103 and 8 for C4, due to memory constraints.

Original: Unless stated otherwise, we train for 100K steps on WikiText-103 and 50K steps for
Revised: Unless stated otherwise, we train for 100K steps on WikiText-103 and 50K steps for

Original: C4.
Revised: C4.

Original: Learning rate In all experiments we use a linear learning rate warm-up period of 4000 steps which increases from 0 to a maximum value (which is tunable hyperparameter).
Revised: Learning rate In all experiments we use a linear learning rate warm-up period of 4000 steps which increases from 0 to a maximum value (which is tunable hyperparameter).

Original: In the remaining iterations, we use a “one-cycle” cosine learning rate schedule (Loshchilov & Hutter, 2016) which reaches 0 at the final iteration.
Revised: In the remaining iterations, we use a “one-cycle” cosine learning rate schedule (Loshchilov & Hutter, 2016) which reaches 0 at the final iteration.

Original: The maximum learning rate was tuned in { 1 , 2 , 3 , 5 } × 10 − n over n ∈ Z for all experiments.
Revised: The maximum learning rate was tuned in { 1 , 2 , 3 , 5 } × 10 − n over n ∈ Z for all experiments.

Original: As the embedding layer does not change across the different depths and skip/normalisation settings we considered, we use a different, fixed maximum learning rate for the embedding layer, which was chosen to match the optimal learning rate for the default transformer ( 2 × 10 − 4 for WikiText-103 and 5 × 10 − 4 for C4) and not tuned further.
Revised: As the embedding layer does not change across the different depths and skip/normalisation settings we considered, we use a different, fixed maximum learning rate for the embedding layer, which was chosen to match the optimal learning rate for the default transformer ( 2 × 10 − 4 for WikiText-103 and 5 × 10 − 4 for C4) and not tuned further.

Original: H
Revised: H. A DDITIONAL IMPLEMENTATION DETAILS FOR INDIVIDUAL EXPERIMENTS

Original: A DDITIONAL IMPLEMENTATION DETAILS FOR INDIVIDUAL EXPERIMENTS

Original: Attention-only kernel matrix evolution: Figs.
Revised: Attention-only kernel matrix evolution: Figs.

Original: 1 and 7 We calculate the kernel matrix Σ evolution directly in T × T kernel matrix-space, where T = 100 .
Revised: 1 and 7 We calculate the kernel matrix Σ evolution directly in T × T kernel matrix-space, where T = 100 .

Original: Our input kernel matrix Σ is sampled assuming a fraction r = 0 .
Revised: Our input kernel matrix Σ is sampled assuming a fraction r = 0 .

Original: 02 of repeated tokens, with value of 1 if the token is repeated, andelse.
Revised: 02 of repeated tokens, with value of 1 if the token is repeated, andelse.

Original: For all configurations of skip/normalisation/attention modifications corresponding to a row of
Revised: For all configurations of skip/normalisation/attention modifications corresponding to a row of

Original: Fig.
Revised: Fig.

Original: 7, we use 8 heads.
Revised: 7, we use 8 heads.

Original: We now detail how each operation in any configuration of Fig.
Revised: We now detail how each operation in any configuration of Fig.

Original: 7 affects the kernel matrix.
Revised: 7 affects the kernel matrix.

Original: From this it should be possible to reconstruct the kernel evolution for any row in Fig.
Revised: From this it should be possible to reconstruct the kernel evolution for any row in Fig.

Original: The 3 possible operations are: 1) attention, 2) skip connection, or 3) LN/RMSNorm operation.
Revised: The 3 possible operations are: 1) attention, 2) skip connection, or 3) LN/RMSNorm operation.

Original: Attention Because our SPA methods (second and third rows) are agnostic to the number of heads at initialisation (all attention matrices in a self-attention block are the same across heads at initialisation), we apply Eq.
Revised: Attention Because our SPA methods (second and third rows) are agnostic to the number of heads at initialisation (all attention matrices in a self-attention block are the same across heads at initialisation), we apply Eq.

Original: (6) directly, so that a single attention block amounts to Σ ← A Σ 
Revised: (6) directly, so that a single attention block amounts to Σ ← A Σ A

Original: A ⊤ for attention matrix A and incoming kernel matrix Σ .
Revised: ⊤ for attention matrix A and incoming kernel matrix Σ .

Original: Our E-SPA method uses γ L = 0 .
Revised: Our E -SPA method uses γ L = 0 .

Revised: 005 and

Original: 005 and our U-SPA method uses ρ L = 0 . 8 .
Revised: our U-SPA method uses ρ L = 0 . 8 .

Original: For all other rows, the self-attention operation uses ALiBi (Press et al., 2022), a popular positional encoder which uses head-dependent pre-softmax bias matrices.
Revised: For all other rows, the self-attention operation uses ALiBi (Press et al., 2022), a popular positional encoder which uses head-dependent pre-softmax bias matrices.

Original: More specifically, from the default pre-softmax bias matrices given by ALiBi (for 8 heads), we obtain 8 attention matrices { A h } 8 h =1 using the softmax operation (which is exact assuming zero query-key dot product at initialisation).
Revised: More specifically, from the default pre-softmax bias matrices given by ALiBi (for 8 heads), we obtain 8 attention matrices { A h } 8 h =1 using the softmax operation (which is exact assuming zero query-key dot product at initialisation).

Original: Because the different heads in transformers are typically concatenated along 8 equal size fractions of the total width d , the kernel evolution[Table] of an attention block on kernel matrix Σ with 8-head ALiBi corresponds to Martens et al.
Revised: Because the different heads in transformers are typically concatenated along 8 equal size fractions of the total width d , the kernel evolution of an attention block on kernel matrix Σ with 8-head ALiBi corresponds to Martens et al.

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] For a skip connection with shortcut weight α and residual weight β , if Σ attn (Σ) denotes the output of a kernel matrix Σ after an self-attention operation (i
Revised: For a skip connection with shortcut weight α and residual weight β , if Σ attn (Σ) denotes the output of a kernel matrix Σ after an self-attention operation (i

Original: [Table]

Original: e. from point 1.
Revised: e. from point 1.

Original: above), then an incoming kernel matrix Σ gets mapped to:
Revised: above), then an incoming kernel matrix Σ gets mapped to:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: For a normalisation operation, the incoming kernel matrix Σ gets mapped to:
Revised: For a normalisation operation, the incoming kernel matrix Σ gets mapped to:

Original: [Equation]
Revised: [Equation]

Original: Hyperparameter tuning of skipless transformers For experiments on WikiText-103 with 100K steps, for our U-SPA transformers we tuned ρ L ∈ { 0 . 6 , 0 . 8 } , and for our E-SPA transformers we tuned γ L ∈ { 0 . 005 , 0 . 2 } .
Revised: Hyperparameter tuning of skipless transformers For experiments on WikiText-103 with 100K steps, for our U-SPA transformers we tuned ρ L ∈ { 0 . 6 , 0 . 8 } , and for our E-SPA transformers we tuned γ L ∈ { 0 . 005 , 0 . 2 } .

Original: For all other settings (longer/deeper training on WikiText-103 or any C experiment), we used the default γ L = 0 .
Revised: For all other settings (longer/deeper training on WikiText-103 or any C experiment), we used the default γ L = 0 .

Original: 005 and ρ L = 0 .
Revised: 005 and ρ L = 0 .

Original: All hyperparameters throughout our work are tuned based on training loss.
Revised: All hyperparameters throughout our work are tuned based on training loss.

Original:  Hyperparameter tuning of transformers with normalised skips All experiments with skip connections (i.e. shortcut weight α ̸ = 0 for either the MLP or attention block) use untransformed GeLU activation in the MLP blocks.
Revised: [Table] Hyperparameter tuning of transformers with normalised skips All experiments with skip connections (i [Table] e. shortcut weight α ̸ = 0 for either the MLP or attention block) use untransformed GeLU activation in the MLP blocks.

Original: We combine E-SPA with normalised skips as described in Appendix F.
Revised: We combine E-SPA with normalised skips as described in Appendix F.

Original: We note that for high shortcut weights α and small final decay rate γ L , then the value of attention matrix diagonal λ
Revised: We note that for high shortcut weights α and small final decay rate γ L , then the value of attention matrix diagonal λ

Original: α , Eq.
Revised: α , Eq.

Original: (32), may not be real.
Revised: (32), may not be real.

Original: This is because the input to the square root in
Revised: This is because the input to the square root in

Original: Eq.
Revised: Eq.

Original: (32) may be negative.
Revised: (32) may be negative.

Original: To get by this, we tune γ L ∈ { 0 . 005 , 0 . 2 , 0 . 4 , 0 . 6 } when using a 36 block transformer on WikiText-103 and γ L ∈ { 0 . 005 , 0 . 2 , 0 . 4 , 0 .
Revised: To get by this, we tune γ L ∈ { 0 . 005 , 0 . 2 , 0 . 4 , 0 . 6 } when using a 36 block transformer on WikiText-103 and γ L ∈ { 0 . 005 , 0 . 2 , 0 . 4 , 0 .

Original: 7 , 1 } for Table 2, which used a 32 block transformer on C4.
Revised: 7 , 1 } for Table 2, which used a 32 block transformer on C4.

Original: For Table 2, the normalised skip connections had separately tuned attention and MLP shortcut weights, as we observed a difference in the optimal shortcut weight for selfattention vs MLP blocks in Fig.
Revised: For Table 2, the normalised skip connections had separately tuned attention and MLP shortcut weights, as we observed a difference in the optimal shortcut weight for selfattention vs MLP blocks in Fig.

Original: We tuned the attention shortcut weight in the range α ∈ { 0 . 98 , 0 . 99 , 0 .
Revised: We tuned the attention shortcut weight in the range α ∈ { 0 . 98 , 0 . 99 , 0 .

Original: 995 , 0 . 9975 , 0 . 999 } and the MLP shortcut weight in the range α ∈ { 0 . 98 , 0 . 99 , 0 . 995 } .
Revised: 995 , 0 . 9975 , 0 . 999 } and the MLP shortcut weight in the range α ∈ { 0 . 98 , 0 . 99 , 0 . 995 } .

Original: Likewise, both the stable residual weights were tuned in β ∈ { 0 . 05 , 0 . 1 , 0 . 15 , 0 . 2 } separately for self-attention and MLP skips.
Revised: Likewise, both the stable residual weights were tuned in β ∈ { 0 . 05 , 0 . 1 , 0 . 15 , 0 . 2 } separately for self-attention and MLP skips.

Original: The selected shortcut/residual weights (using validation performance) are presented in Table 7.
Revised: The selected shortcut/residual weights (using validation performance) are presented in Table 7.

Original: Depth scaling Due to memory constraints, for our deeper networks in Table 3 we use width d = 512 , with 8 heads to give d k = 64 .
Revised: Depth scaling Due to memory constraints, for our deeper networks in Table 3 we use width d = 512 , with 8 heads to give d k = 64 .

Original: All depth scaling runs used a DKS-transformed GeLU with ξ = 5 .
Revised: All depth scaling runs used a DKS-transformed GeLU with ξ = 5 .

Original: I T HEORETICAL RESULTS
Revised: I T HEORETICAL RESULTS

Original: In this section we state and prove our theoretical results, including Theorems 1 and 2:
Revised: In this section we state and prove our theoretical results, including Theorems 1 and 2:

Original: Theorem 1.
Revised: Theorem 1.

Original: (Non-negativity for U-SPA) Let Σ=(1 − ρ ) I T + ρ 11 ⊤ and Σ ′ =(1 − ρ ′ )
Revised: (Non-negativity for U-SPA) Let Σ=(1 − ρ ) I T + ρ 11 ⊤ and Σ ′ =(1 − ρ ′ )

Original: I T + ρ ′ 11 ⊤ , with respective (positive) Cholesky factors L and L ′ .
Revised: I T + ρ ′ 11 ⊤ , with respective (positive) Cholesky factors L and L ′ .

Original: Then if ρ ≤ ρ ′ , we have L ′ L − 1 is elementwise non-negative.
Revised: Then if ρ ≤ ρ ′ , we have L ′ L − 1 is elementwise non-negative.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Non-negativity for E-SPA) Let matrices (Σ) i,j = exp ( − γ | i − j | )) and (Σ ′ )
Revised: (Non-negativity for E-SPA) Let matrices (Σ) i,j = exp ( − γ | i − j | )) and (Σ ′ )

Original: i,j = exp ( − γ ′
Revised: i,j = exp ( − γ ′

Original: |
Revised: |

Original: i − j | )) with respective (positive) Cholesky factors L and L ′ .
Revised: i − j | )) with respective (positive) Cholesky factors L and L ′ .

Original: Then if γ ≥ γ ′ , we have
Revised: Then if γ ≥ γ ′ , we have

Original: L ′ L − 1 is elementwise non-negative.
Revised: L ′ L − 1 is elementwise non-negative.

Original: We prove Theorem 1 second as it is more involved.
Revised: We prove Theorem 1 second as it is more involved.

Original: I. P ROOF OF T HEOREM
Revised: I. P ROOF OF T HEOREM

Original: We actually prove Theorem 2 as a corollary of Theorem 3, which provides the analytic form for
Revised: We actually prove Theorem 2 as a corollary of Theorem 3, which provides the analytic form for

Original: L ′ L − 1 .
Revised: L ′ L − 1 .

Original: Theorem 3.
Revised: Theorem 3.

Original: Let matrices (Σ) i,j = exp ( − γ | i − j | )) and (Σ ′ )
Revised: Let matrices (Σ) i,j = exp ( − γ | i − j | )) and (Σ ′ )

Original: i,j = exp ( − γ ′
Revised: i,j = exp ( − γ ′

Original: |
Revised: |

Original: i − j | )) with respective (positive) Cholesky factors L and L ′ .
Revised: i − j | )) with respective (positive) Cholesky factors L and L ′ .

Original: Then if γ ≥ γ ′ > 0 , we have A = L ′ L − 1 takes the following form:
Revised: Then if γ ≥ γ ′ > 0 , we have A = L ′ L − 1 takes the following form:

Original: [Equation]
Revised: [Equation]

Original: if i
Revised: else

Original: [Equation]
Revised: [Equation]

Original: j [Equation] if i [Equation]
Revised: where a ( γ )

Original: j ̸

Original: [Equation] [Equation] else (36) where a ( γ ) = (cid:112) 1 − exp ( − 2 γ ) , and likewise a ( γ ′ ) =
Revised: = (cid:112) 1 − exp ( − 2 γ ) , and likewise a ( γ ′ ) =

Original: (cid:112) 1 − exp ( − 2 γ ′ )
Revised: (cid:112) 1 − exp ( − 2 γ ′ )

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: From Theorem 3, we have the analytic form of A .
Revised: From Theorem 3, we have the analytic form of A .

Original: Clearly a ( γ ) , a ( γ ′ ) are positive, and moreover if γ ′ ≤ γ , then exp ( − γ ′ ) − exp ( − γ ) is non-negative.
Revised: Clearly a ( γ ) , a ( γ ′ ) are positive, and moreover if γ ′ ≤ γ , then exp ( − γ ′ ) − exp ( − γ ) is non-negative.

Original: Finally, because a ( γ ′ ) a ( γ ) ≤ 1 when γ ≥ γ ′ , then exp ( − γ ′ ) − a ( γ ′ ) a ( γ ) exp ( − γ ) is non-negative too.
Revised: Finally, because a ( γ ′ ) a ( γ ) ≤ 1 when γ ≥ γ ′ , then exp ( − γ ′ ) − a ( γ ′ ) a ( γ ) exp ( − γ ) is non-negative too.

Original: To prove Theorem 3, we first compute what the analytic form of Cholesky factor L for (Σ) i,j = exp ( − γ |
Revised: To prove Theorem 3, we first compute what the analytic form of Cholesky factor L for (Σ) i,j = exp ( − γ |

Original: i − j | )) takes in Lemma 1.
Revised: i − j | )) takes in Lemma 1.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let (Σ) i,j = exp ( − γ |
Revised: Let (Σ) i,j = exp ( − γ |

Original: i − j | )) with (positive) Cholesky factors L such that LL ⊤ = Σ .
Revised: i − j | )) with (positive) Cholesky factors L such that LL ⊤ = Σ .

Original: Then, we have:
Revised: Then, we have:

Revised: [Equation]

Revised: else

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: It is clear that Σ is positive semi definite, as it is the covariance matrix of a stationary Ornstein-Uhlenbeck process, hence a Cholesky factor must exist.
Revised: It is clear that Σ is positive semi definite, as it is the covariance matrix of a stationary Ornstein-Uhlenbeck process, hence a Cholesky factor must exist.

Original: We now show that it is
Revised: We now show that it is

Original: L , Eq.
Revised: L , Eq.

Original: If we define l = min ( m, n ) , then we have:
Revised: If we define l = min ( m, n ) , then we have:

Original: [Equation]
Revised: [Equation]

Original: By the uniqueness of (positive) Cholesky factors, the proof is complete.
Revised: By the uniqueness of (positive) Cholesky factors, the proof is complete.

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: We want to show ( AL ) k,l = ( L ′ )
Revised: We want to show ( AL ) k,l = ( L ′ )

Original: k,l , ∀ k, l .
Revised: k,l , ∀ k, l .

Original: This is clearly true for the top diagonal k = l = 1 .
Revised: This is clearly true for the top diagonal k = l = 1 .

Original: We now show this for the rest of the first column, when l = 1 , k > 1 :
Revised: We now show this for the rest of the first column, when l = 1 , k > 1 :

Original: [Equation]
Revised: [Equation]

Original: AL
Revised: Applying the geometric sum to Eq. (49) yields:

Original: [Equation]
Revised: [Equation]

Original: k, 1 = A k, 1 + A k,k L k, 1 + k
Revised: −

Original: [Equation]
Revised: [Equation]

Original: i =

Original: [Equation]
Revised: [Equation]

Original: A k

Original: [Equation][Equation] Applying the geometric sum to Eq. (49) yields:
Revised: [Equation]

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: γ ′

Revised: [Equation]

Revised: [Equation]

Revised: (54) =

Revised: exp ( − γ ′

Revised: ( k − 1))

Revised: (55) + a ( γ ′ ) a ( γ )

Original: as desired.
Revised: [Equation] [Equation] [Equation] [Equation]

Original: Likewise, if l > 1 , l ≤ k : ( AL ) k,l =
Revised: ) (cid:1)(cid:1)(cid:3) (56) = exp ( − γ ′ ( k − 1)) (57) + a ( γ ′ ) a ( γ ) (cid:2) − exp (cid:0) − γ ′ ( k − 2) − γ (cid:1) + exp (cid:0) − γ ′

Original: A k,k L k,l + 
Revised: ( k − 2) − γ (cid:1) + exp ( − γ ( k − 1)) − exp ( − γ ( k − 1))] (58) = exp ( − γ ′ ( k − 1))

Original: [Equation]
Revised: (59) =

Original: [Equation]
Revised: L ′ k,(60) as desired. Likewise, if l >

Original: [Equation]
Revised: [Equation]

Revised: l ≤ k

Original: [Equation]
Revised: [Equation]

Original: I. P ROOF OF T HEOREM
Revised: I. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: (Non-negativity for U-SPA) Let Σ=(1 − ρ ) I T + ρ 11 ⊤ and Σ ′ =(1 − ρ ′ )
Revised: (Non-negativity for U-SPA) Let Σ=(1 − ρ ) I T + ρ 11 ⊤ and Σ ′ =(1 − ρ ′ )

Original: I T + ρ ′ 11 ⊤ , with respective (positive) Cholesky factors L and L ′ .
Revised: I T + ρ ′ 11 ⊤ , with respective (positive) Cholesky factors L and L ′ .

Original: Then if ρ ≤ ρ ′ , we have L ′ L − 1 is elementwise non-negative.
Revised: Then if ρ ≤ ρ ′ , we have L ′ L − 1 is elementwise non-negative.

Original: I.2. P RELIMINARIES
Revised: I.2. P RELIMINARIES

Original: Before diving into the actual proof of the theorem we will derive several useful properties and notations.
Revised: Before diving into the actual proof of the theorem we will derive several useful properties and notations.

Original: First we make a slight notational change from the main text of the theorem by replacing Σ , which depends on ρ , with C n ( x ) , where n represents the size of the matrix, while x replaces ρ .
Revised: First we make a slight notational change from the main text of the theorem by replacing Σ , which depends on ρ , with C n ( x ) , where n represents the size of the matrix, while x replaces ρ .

Original: In addition, since the case of ρ = ρ ′ (or x = y in the rest of the proof) is trivial, since then the resulting matrix is the identity, we will restrict ourselves to dealing with the case where 0 < x < y < 1 .
Revised: In addition, since the case of ρ = ρ ′ (or x = y in the rest of the proof) is trivial, since then the resulting matrix is the identity, we will restrict ourselves to dealing with the case where 0 < x < y < 1 .

Original: In any of the mathematical derivations we will denote with capital English letters (e.g. A, B, C, T ) any temporary expressions, that will be expanded on the following lines.
Revised: In any of the mathematical derivations we will denote with capital English letters (e.g. A, B, C, T ) any temporary expressions, that will be expanded on the following lines.

Original: Note that these are never general definitions, so they might be used multiple times for different expressions.
Revised: Note that these are never general definitions, so they might be used multiple times for different expressions.

Original: We will denote vectors and vector functions in bold and scalars and scalar function in standard font.
Revised: We will denote vectors and vector functions in bold and scalars and scalar function in standard font.

Original: Let 1 n be the n -dimensional vector with only ones:
Revised: Let 1 n be the n -dimensional vector with only ones:

Original: [Equation]
Revised: [Equation]

Original: We will denote with x n the n -dimensional vector with only x ’s:
Revised: We will denote with x n the n -dimensional vector with only x ’s:

Original: [Equation]
Revised: [Equation]

Original: First, we define the linear map p n ( x ) as:
Revised: First, we define the linear map p n ( x ) as:

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: The vector 1 n is an eigenvector of C n ( x ) with an eigenvalue p n − 1 ( x ) .
Revised: The vector 1 n is an eigenvector of C n ( x ) with an eigenvalue p n − 1 ( x ) .

Original: Proof.
Revised: Proof.

Original: Directly calculating the k -th entry of the product C n ( x ) 1 n gives:
Revised: Directly calculating the k -th entry of the product C n ( x ) 1 n gives:

Original: [Equation]
Revised: [Equation]

Original: which directly implies that:
Revised: which directly implies that:

Original: [Equation]
Revised: [Equation]

Original: Corollary 1.
Revised: Corollary 1.

Original: The vector 1 n is an eigenvector of C n ( x ) − 1 with an eigenvalue 1 p n − 1 ( x ) .
Revised: The vector 1 n is an eigenvector of C n ( x ) − 1 with an eigenvalue 1 p n − 1 ( x ) .

Original: Corollary 2.
Revised: Corollary 2.

Original: C n ( x ) − 1 x n = 1 p n − 1 ( x ) x n .
Revised: C n ( x ) − 1 x n = 1 p n − 1 ( x ) x n .

Original: Corollary 3.
Revised: Corollary 3.

Original: x Tn C n ( x ) − 1 x n =nx
Revised: x Tn C n ( x ) − 1 x n =

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Further we define the following useful functions:
Revised: Further we define the following useful functions:

Original: [Equation]
Revised: [Equation]

Original: Definition 1.
Revised: Definition 1.

Original: The function v n : I [0 , 1] × I [0 , 1] → R n maps the two bounded scalars x and y to the vector:
Revised: The function v n : I [0 , 1] × I [0 , 1] → R n maps the two bounded scalars x and y to the vector:

Original: [Equation]
Revised: [Equation]

Original: I.2.
Revised: I.2.

Original: L EMMAS
Revised: L EMMAS

Original: Lemma 2.
Revised: Lemma 2.

Original: If 0 < x < y < 1 then all entries of the vector v n ( x, y ) are non-negative - [ v n ( x, y )]
Revised: If 0 < x < y < 1 then all entries of the vector v n ( x, y ) are non-negative - [ v n ( x, y )]

Original: i ⩾ 0 .
Revised: i ⩾ 0 .

Original: Lemma 3.
Revised: Lemma 3.

Original: If 0 < x < y < 1 then the function g n ( x, y ) =
Revised: If 0 < x < y < 1 then the function g n ( x, y ) =

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: is non-negative.
Revised: is non-negative.

Original: Lemma 4.
Revised: Lemma 4.

Original: If 0 < x < y < 1 then the function f n (
Revised: If 0 < x < y < 1 then the function f n (

Original: [Equation]
Revised: [Equation]

Original: Definition 2.
Revised: Definition 2.

Original: The partial sum the functions f n ( x, y ) from k +1 to n − 1 will be denote by h nk ( x, y )
Revised: The partial sum the functions f n ( x, y ) from k +1 to n − 1 will be denote by h nk ( x, y )

Original: = (cid:80) n −i =
Revised: = (cid:80) n −i =

Original: k +

Revised: k +

Original: f i ( x, y ) , which from Lemma 4 follow are always negative.
Revised: f i ( x, y ) , which from Lemma 4 follow are always negative.

Original: I.2. M AIN PROOF
Revised: I.2. M AIN PROOF

Original: The theorem is proven by induction.
Revised: The theorem is proven by induction.

Original: First for n = 1 we have that C 1 ( x ) =
Revised: First for n = 1 we have that C 1 ( x ) =

Original: C 1 ( y ) =
Revised: C 1 ( y ) =

Original: [1] , which implies that L 1 ( x ) =
Revised: [1] , which implies that L 1 ( x ) =

Original: L 1 ( y ) =
Revised: L 1 ( y ) =

Original: [1] and the condition is trivially satisfied.
Revised: [1] and the condition is trivially satisfied.

Original: Now assuming that the statement is true for all integers up to n , we will prove that it also holds for n + 1 :
Revised: Now assuming that the statement is true for all integers up to n , we will prove that it also holds for n + 1 :

Original: [Equation]
Revised: [Equation]

Original: T = (cid:20)
Revised: T = (cid:20)

Original: L n ( x ) 0 n x Tn L n ( x ) − T (cid:112) d n ( x ) (cid:21) (cid:20) L n ( x ) T
Revised: L n ( x ) 0 n x Tn L n ( x ) − T (cid:112) d n ( x ) (cid:21) (cid:20) L n ( x ) T

Original: L n ( x ) − 1 x n 0 Tn (cid:112) d n ( x ) (cid:21) =
Revised: L n ( x ) − 1 x n 0 Tn (cid:112) d n ( x ) (cid:21) =

Original: (cid:20) L n ( x ) L n ( x ) T x n x Tn
Revised: (cid:20) L n ( x ) L n ( x ) T x n x Tn

Original: A (cid:21)
Revised: A (cid:21)

Original: A = x Tn L n ( x ) − T L n ( x ) − 1 x n + d n ( x ) = x
Revised: A = x Tn L n ( x ) − T L n ( x ) − 1 x n + d n ( x ) = x

Original: Tn C n ( x ) − 1 x n + d n ( x ) =
Revised: Tn C n ( x ) − 1 x n + d n ( x ) =

Original: L n +1 ( x ) −= (cid:34) L n ( x ) −0 n − 1 r n ( x ) x Tn 1 √ d n ( x ) (cid:35)
Revised: L n +1 ( x ) −= (cid:34) L n ( x ) −0 n − 1 r n ( x ) x Tn 1 √ d n ( x ) (cid:35)

Original: L n +1 ( x ) L n +1 ( x ) −= (cid:20)
Revised: L n +1 ( x ) L n +1 ( x ) −= (cid:20)

Original: L n ( x ) 0 n x Tn L n ( x ) − T (cid:112) d n ( x ) (cid:21) (cid:34) L n ( x ) −0 n − 1 r n ( x ) x Tn 1 √ d n ( x ) (cid:35) = (cid:34)
Revised: L n ( x ) 0 n x Tn L n ( x ) − T (cid:112) d n ( x ) (cid:21) (cid:34) L n ( x ) −0 n − 1 r n ( x ) x Tn 1 √ d n ( x ) (cid:35) = (cid:34)

Original: I n 0 n x Tn L n ( x ) − T L n ( x ) − 1 − √ d n ( x ) r n ( x ) x Tn (cid:35) =
Revised: I n 0 n x Tn L n ( x ) − T L n ( x ) − 1 − √ d n ( x ) r n ( x ) x Tn (cid:35) =

Original: (cid:20)
Revised: (cid:20)

Original: I n 0 n x Tn C n ( x ) − 1 − 1 p n − 1 ( x ) x Tn (cid:21) =
Revised: I n 0 n x Tn C n ( x ) − 1 − 1 p n − 1 ( x ) x Tn (cid:21) =

Original: (cid:20) I n 0 n 

Revised: (cid:20) I n 0 n

Original: Tn (cid:21)
Revised: Tn (cid:21)

Original: L n +1 ( y ) L n +1 ( x ) −= (cid:20)
Revised: L n +1 ( y ) L n +1 ( x ) −= (cid:20)

Original: L n ( y ) 0 n y Tn L n ( y ) − T (cid:112) d n ( y ) (cid:21) (cid:34) L n (
Revised: L n ( y ) 0 n y Tn L n ( y ) − T (cid:112) d n ( y ) (cid:21) (cid:34) L n (

Original: [Equation]
Revised: [Equation]

Original: Using the fact that L n ( y ) L n ( x ) − 1 is a lower triangular and non-negative matrix by the inductive assumption combined with Lemma 2 and the fact that d n ( x ) ⩾ 

Revised: Using the fact that L n ( y ) L n ( x ) − 1 is a lower triangular and non-negative matrix by the inductive assumption combined with Lemma 2 and the fact that d n ( x ) ⩾

Original: it follows that L n +1 ( y ) L n +1 ( x ) −is also a lower triangular non-negative matrix.
Revised: it follows that L n +1 ( y ) L n +1 ( x ) −is also a lower triangular non-negative matrix.

Original: First we will inspect the evolution of v n ( x, y ) as we increase n :
Revised: First we will inspect the evolution of v n ( x, y ) as we increase n :

Original: [Equation]
Revised: [Equation]

Original: n ( y ) − 1 y n y (1 − y )
Revised: n ( y ) − 1 y n y (1 − y )

Original: r n ( y )
Revised: r n ( y )

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 
Revised: 

Original:  v n +
Revised:  v n +

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y )
Revised: y )

Original: =
Revised: =

Original: A −
Revised: A −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x ) 1 p n
Revised: x ) 1 p n

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: d n +
Revised: d n +

Original: [Equation]
Revised: [Equation]

Original: y )
Revised: y )

Original: d n +
Revised: d n +

Original: [Equation]
Revised: [Equation]

Original: x ) x p n ( x )
Revised: x ) x p n ( x )

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y )
Revised: y )

Original: [Equation]
Revised: [Equation]

Original: y (1 − y )
Revised: y (1 − y )

Original: r n ( x ) r n ( y )
Revised: r n ( x ) r n ( y )

Original: [Equation]
Revised: [Equation]

Original: d n +1 ( y ) d n +1 ( x ) 1 p n ( x ) −
Revised: d n +1 ( y ) d n +1 ( x ) 1 p n ( x ) −

Original: [Table]
Revised: [Table]

Original: Using the definition of from Lemma
Revised: Using the definition of from Lemma

Original: [Equation]
Revised: [Equation]

Original: Lemma 4 and Definition 2 we have that:
Revised: Lemma 4 and Definition 2 we have that:

Original: [Equation]
Revised: [Equation]

Original: Or in other words we have that:
Revised: Or in other words we have that:

Original: [Equation]
Revised: [Equation]

Original: Thus proving the lemma reduces to proving that:
Revised: Thus proving the lemma reduces to proving that:

Original: [Equation]
Revised: [Equation]

Original: Expanding on the equation that we need to prove is positive:
Revised: Expanding on the equation that we need to prove is positive:

Original: [Equation]
Revised: [Equation]

Original: = y (1 − y )
Revised: = y (1 − y )

Original: p k − 1 ( x ) r k ( x ) r k
Revised: p k − 1 ( x ) r k ( x ) r k

Original: ( y ) − n (cid:88) i =
Revised: ( y ) − n (cid:88) i =

Original: k + xy (1 − y )
Revised: k + xy (1 − y )

Original: r i ( x ) r i ( y )
Revised: r i ( x ) r i ( y )

Original: − B
Revised: − B

Original: B = (cid:115)
Revised: B = (cid:115)

Original: d k +1 ( y )
Revised: d k +1 ( y )

Original: d k +1 ( x ) x p k
Revised: d k +1 ( x ) x p k

Original: ( x ) + n (cid:88)
Revised: ( x ) + n (cid:88)

Original: i =
Revised: i =

Original: k + (cid:115)
Revised: k + (cid:115)

Original: d i +1 ( y )
Revised: d i +1 ( y )

Original: d i +1 ( x ) x
Revised: d i +1 ( x ) x

Original: p
Revised: p

Original: i ( x ) − n (cid:88) i =
Revised: i ( x ) − n (cid:88) i =

Original: k + (cid:115)
Revised: k + (cid:115)

Original: d i ( y )
Revised: d i ( y )

Original: d i ( x ) x p i − 1 ( x ) = n (cid:88) i
Revised: d i ( x ) x p i − 1 ( x ) = n (cid:88) i

Original: = k (cid:115)
Revised: = k (cid:115)

Original: d i +1 ( y )
Revised: d i +1 ( y )

Original: d i +1 ( x ) x
Revised: d i +1 ( x ) x

Original: p
Revised: p

Original: i ( x ) − n − 1 (cid:88)
Revised: i ( x ) − n − 1 (cid:88)

Original: i =
Revised: i =

Original: k (cid:115)
Revised: k (cid:115)

Original: d i +1 ( y )
Revised: d i +1 ( y )

Original: d i +1 ( x ) x
Revised: d i +1 ( x ) x

Original: p
Revised: p

Original: i ( x ) =
Revised: i ( x ) =

Original: (cid:115)
Revised: (cid:115)

Original: d n +1 ( y )
Revised: d n +1 ( y )

Original: d n +1 ( x ) x p n ( x )
Revised: d n +1 ( x ) x p n ( x )

Original: L nk ( x, y ) =
Revised: L nk ( x, y ) =

Original: y (1 − y )
Revised: y (1 − y )

Original: p k − 1 ( x ) r k ( x ) r k
Revised: p k − 1 ( x ) r k ( x ) r k

Original: ( y ) − n (cid:88) i =
Revised: ( y ) − n (cid:88) i =

Original: k + xy (1 − y )
Revised: k + xy (1 − y )

Original: r i ( x ) r i ( y ) − (cid:115)
Revised: r i ( x ) r i ( y ) − (cid:115)

Original: d n +1 ( y )
Revised: d n +1 ( y )

Original: d n +1 ( x ) x p n ( x )
Revised: d n +1 ( x ) x p n ( x )

Original: Now we turn our attention to the sum in the middle:
Revised: Now we turn our attention to the sum in the middle:

Original: B =
Revised: B =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) n
Revised: ) n

Original: [Equation]
Revised: [Equation]

Original: i = k
Revised: i = k

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ( x )
Revised: ( x )

Original: [Equation]
Revised: [Equation]

Original: i ( y ) =
Revised: i ( y ) =

Original: xy (cid:114) 1 − y 1 − x n (cid:88) i =
Revised: xy (cid:114) 1 − y 1 − x n (cid:88) i =

Original: k
Revised: k

Original: + (cid:112) (1 − x )(1 − y )
Revised: + (cid:112) (1 − x )(1 − y )

Original: r
Revised: r

Original: i ( x ) r i ( y )
Revised: i ( x ) r i ( y )

Original: Using Cacuhy − Schwartz inequality :
Revised: Using Cacuhy − Schwartz inequality :

Original: [Equation]
Revised: [Equation]

Original: i 
Revised: i = k

Original: [Equation]
Revised: [Equation]

Original:  (1 − x )(1 − y )
Revised: (cid:112) (1 − x )(1 − y )

Original: r i ( x ) r i ( y ) (cid:33)⩽ (cid:32) n (cid:88) i
Revised: r i ( x ) r i ( y ) (cid:33)⩽ (cid:32) n (cid:88) i

Original: = k
Revised: = k

Original: + 1 − x r i ( x )(cid:33) (cid:32) n (cid:88)
Revised: + 1 − x r i ( x )(cid:33) (cid:32) n (cid:88)

Original: i = k
Revised: i = k

Original: + 1 − y r
Revised: + 1 − y r

Original: i
Revised: i

Original: ( y )(cid:33)
Revised: ( y )(cid:33)

Original: Let’s define the partial sum in the brackets as:
Revised: Let’s define the partial sum in the brackets as:

Original: [Equation]
Revised: [Equation]

Original: We will prove by induction that:
Revised: We will prove by induction that:

Original: [Equation]
Revised: [Equation]

Original: First for n =
Revised: First for n =

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: we have that I nn ( x ) =
Revised: we have that I nn ( x ) =

Original: [Equation]
Revised: [Equation]

Original: 1 ( x ) which clearly satisfies the above quation.
Revised: 1 ( x ) which clearly satisfies the above quation.

Original: Assuming this is true for n , we will now show it holds for n + 1 :
Revised: Assuming this is true for n , we will now show it holds for n + 1 :

Original: I n +1 k
Revised: I n +1 k

Original: [Equation]
Revised: [Equation]

Original: Which concludes the proof.
Revised: Which concludes the proof.

Original: This now means that:
Revised: This now means that:

Original: [Equation]
Revised: [Equation]

Original: Thus we can conclude that
Revised: Thus we can conclude that

Original: [Equation]
Revised: [Equation]

Original: From the fact that h nk ( x, y ) ⩾ 

Revised: From the fact that h nk ( x, y ) ⩾

Original: it impliest that L nk is a decreasing function of n , hence to we only need to show that for any fixed x and y L ∞ k ( x, y ) is positive.
Revised: it impliest that L nk is a decreasing function of n , hence to we only need to show that for any fixed x and y L ∞ k ( x, y ) is positive.

Original: For this
Revised: For this

Original: [Equation]
Revised: [Equation]

Original: we need to take the limit of the second and third term in the above equation.
Revised: we need to take the limit of the second and third term in the above equation.

Original: [Equation]
Revised: [Equation]

Original: where the last line is true since the denominator is of higher degree in n .
Revised: where the last line is true since the denominator is of higher degree in n .

Original: Taking the limit of the second term corresponds to computing the limit:
Revised: Taking the limit of the second term corresponds to computing the limit:

Original: [Equation]
Revised: [Equation]

Original: Hence, this means that:
Revised: Hence, this means that:

Original: [Equation]
Revised: [Equation]

Original: Note:
Revised: Note:

Original: All of the limits must be taken for fixed x and y
Revised: All of the limits must be taken for fixed x and y

Original: (e.g. we can’t have them approach 0 or 1 simultanously with n ).
Revised: (e.g. we can’t have them approach 0 or 1 simultanously with n ).

Original: [Equation]
Revised: [Equation]

Original: ( n − k )p k ( x ) p n ( x ) p k ( y ) p n ( y ) − (cid:115)
Revised: ( n − k )p k ( x ) p n ( x ) p k ( y ) p n ( y ) − (cid:115)

Original: d n +1 ( y )
Revised: d n +1 ( y )

Original: d n +1 ( x ) x p n ( x ) =
Revised: d n +1 ( x ) x p n ( x ) =

Original: A − ( B + C )
Revised: A − ( B + C )

Original: A =
Revised: A =

Original: y (1 − y )
Revised: y (1 − y )

Original: p k − 1 ( x ) r k ( x ) r k
Revised: p k − 1 ( x ) r k ( x ) r k

Original: ( y ) =
Revised: ( y ) =

Original: y (1 − y )
Revised: y (1 − y )

Original: p k − 1 ( x ) (cid:112) (1 − x ) p k
Revised: p k − 1 ( x ) (cid:112) (1 − x ) p k

Original: ( x ) p k − 1 ( x ) (cid:112) (1 − y )
Revised: ( x ) p k − 1 ( x ) (cid:112) (1 − y )

Original: p k
Revised: p k

Original: ( y ) p k − 1 ( y ) =
Revised: ( y ) p k − 1 ( y ) =

Original: y (cid:114) 1 − y 1 − x (cid:112) p k
Revised: y (cid:114) 1 − y 1 − x (cid:112) p k

Original: ( x ) p k ( y ) (cid:115) p k − 1 ( x ) p k − 1 ( y )
Revised: ( x ) p k ( y ) (cid:115) p k − 1 ( x ) p k − 1 ( y )

Original: C = (cid:115) d n +1 ( y ) d n +1 ( x ) x p n ( x ) =
Revised: C = (cid:115) d n +1 ( y ) d n +1 ( x ) x p n ( x ) =

Original: (cid:118)(cid:117)(cid:117)(cid:116) (1 − y )
Revised: (cid:118)(cid:117)(cid:117)(cid:116) (1 − y )

Original: p n +1 ( y ) p n ( y ) (1 − x ) p n +1 ( x ) p n ( x ) x p n ( x )
Revised: p n +1 ( y ) p n ( y ) (1 − x ) p n +1 ( x ) p n ( x ) x p n ( x )

Original: = x (cid:115) (1 − y )
Revised: = x (cid:115) (1 − y )

Original: p n +1 ( y ) (1 − x ) p n ( y ) p n ( x ) p n +1 ( x ) =
Revised: p n +1 ( y ) (1 − x ) p n ( y ) p n ( x ) p n +1 ( x ) =

Original: x (cid:114) 1 − y 1 − x (cid:115) p n +1 ( y ) p n ( y ) p n ( x ) p n +1 ( x )
Revised: x (cid:114) 1 − y 1 − x (cid:115) p n +1 ( y ) p n ( y ) p n ( x ) p n +1 ( x )

Original: = x (cid:114) 1 − y 1 − x (cid:112) p k
Revised: = x (cid:114) 1 − y 1 − x (cid:112) p k

Original: ( x ) p k ( y ) (cid:115)
Revised: ( x ) p k ( y ) (cid:115)

Original: p k
Revised: p k

Original: ( x ) p k ( y ) p n +1 ( y )
Revised: ( x ) p k ( y ) p n +1 ( y )

Original: p n ( x ) p n ( y ) p n +1 ( x )
Revised: p n ( x ) p n ( y ) p n +1 ( x )

Original: A − B − C = (cid:114) 1 − y 1 − x (cid:112) p k
Revised: A − B − C = (cid:114) 1 − y 1 − x (cid:112) p k

Original: ( x ) 
Revised: ( x ) p

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ) p

Original: [Equation]
Revised: [Equation]

Revised: p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y )
Revised: x )

Original: A
Revised: [Equation] p [Equation] [Equation]

Original:  [Equation] x + 1)
Revised: p [Equation] y

Original: [Equation]
Revised: [Equation]

Original: nx
Revised: p n

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] nx
Revised: y )

Original: [Equation]
Revised: A

Original: [Equation]
Revised: [Equation]

Original: x
Revised: y

Original: [Equation]
Revised: [Equation]

Original: y
Revised: ( kx

Original: [Equation]
Revised: [Equation]

Original: ( n − k )( kx − x +  1)
Revised: x + 1)( nx + 1)( ny + 1)

Original: [Equation]
Revised: [Equation]

Original: nx 
Revised: nx + 1) B

Original: [Equation]
Revised: [Equation]

Original: x [Equation] [Equation] x 
Revised: x 2 y 2 ( n − k )( kx − x + 1)( nx

Original: [Equation]
Revised: [Equation]

Original: ( kx + 1)(
Revised: x + 1) C

Original: [Equation]
Revised: [Equation]

Original: [Equation] − y + 1)( ny [Equation] y + 1)
Revised: x 2 ( kx + 1)( ky + 1)( ky − y + 1)( ny + y + 1)

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: R ( C
Revised: R ( C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) =
Revised: ) =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: x p n
Revised: x p n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) p
Revised: ) p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) p
Revised: ) p

Original: [Equation]
Revised: [Equation]

Original: ( x ) p
Revised: ( x ) p

Original: [Equation]
Revised: [Equation]

Original: 1 ( x ) −
Revised: 1 ( x ) −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: 2 R p
Revised: 2 R p

Original: [Equation]
Revised: [Equation]

Original: y )
Revised: y )

Original: K = p n
Revised: K = p n

Original: [Equation]
Revised: [Equation]

Original: 1 ( x )( yk ( n
Revised: 1 ( x )( yk ( n

Original: [Equation]
Revised: [Equation]

Original: ( n
Revised: ( n

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1) − xp n
Revised: 1) − xp n

Original: [Equation]
Revised: [Equation]

Original: 1 ( y ) p
Revised: 1 ( y ) p

Original: [Equation]
Revised: [Equation]

Original: ( x ) p
Revised: ( x ) p

Original: [Equation]
Revised: [Equation]

Original: ( y ) =
Revised: ( y ) =

Original: ( nx
Revised: ( nx

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: 1) y (
Revised: 1) y (

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: nk
Revised: nk

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x ( ny + 1)( kx
Revised: x ( ny + 1)( kx

Original: [Equation]
Revised: [Equation]

Original: 1)( ky + 1)
Revised: 1)( ky + 1)

Original: C − ( A
Revised: C − ( A

Original: [Equation]
Revised: [Equation]

Original: B ) = √ xy (cid:114) 1 − y 1 − x (cid:115)
Revised: B ) = √ xy (cid:114) 1 − y 1 − x (cid:115)

Original: p k
Revised: p k

Original: ( x ) p k ( y ) (cid:32) 1 − √ xy ( n − k ) (cid:112) p n ( x ) p n ( y ) − (cid:115) xp n +1 ( y )
Revised: ( x ) p k ( y ) (cid:32) 1 − √ xy ( n − k ) (cid:112) p n ( x ) p n ( y ) − (cid:115) xp n +1 ( y )

Original: p k ( x ) p k ( y ) yp n ( y ) p n ( x ) p n +1 ( x ) (cid:33) =
Revised: p k ( x ) p k ( y ) yp n ( y ) p n ( x ) p n +1 ( x ) (cid:33) =

Original: √ xy (cid:114) 1 − y 1 − x (cid:115) p k
Revised: √ xy (cid:114) 1 − y 1 − x (cid:115) p k

Original: ( x ) p k ( y ) (cid:18)
Revised: ( x ) p k ( y ) (cid:18)

Original: [Equation]
Revised: [Equation]

Original: I.2. P ROOF OF L EMMA
Revised: I.2. P ROOF OF L EMMA

Original: From the definition of g n ( x, y ) and the fact that p n ( x ) and r n ( x ) are non-negative functions we can conclude that:
Revised: From the definition of g n ( x, y ) and the fact that p n ( x ) and r n ( x ) are non-negative functions we can conclude that:

Original: [Equation]
Revised: [Equation]

Original: Denoting with A the left hand side of the second equation we have:
Revised: Denoting with A the left hand side of the second equation we have:

Original: [Equation]
Revised: [Equation]

Original: Hence for x ⩽ y we have that A ⩾ 0 ⇔ g n ( x, y ) ⩾ 0 .
Revised: Hence for x ⩽ y we have that A ⩾ 0 ⇔ g n ( x, y ) ⩾ 0 .

Original: I.2. P ROOF OF L EMMA
Revised: I.2. P ROOF OF L EMMA

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: First we will prove that C − B ⩾
Revised: First we will prove that C − B ⩾

Original: [Equation]
Revised: [Equation]

Original: ⇒ C ⩾ B
Revised: ⇒ C ⩾ B

Original: With this we can now conclude that f n
Revised: With this we can now conclude that f n

Original: [Equation]
Revised: [Equation]

Original: y )
Revised: y )

Original: [Equation]
Revised: [Equation]