Exact learning dynamics of deep linear networks with prior knowledge
Abstract
Learning in deep neural networks is known to depend critically on the knowledge embedded in the initial network weights.
However, few theoretical results have precisely linked prior knowledge to learning dynamics.
Here we derive exact solutions to the dynamics of learning with rich prior knowledge in deep linear networks by generalising Fukumizu’s matrix Riccati solution [1].
We obtain explicit expressions for the evolving network function, hidden representational similarity, and neural tangent kernel over training for a broad class of initialisations and tasks.
The expressions reveal a class of task-independent initialisations that radically alter learning dynamics from slow non-linear dynamics to fast exponential trajectories while converging to a global optimum with identical representational similarity, dissociating learning trajectories from the structure of initial internal representations.
We characterise how network weights dynamically align with task structure, rigorously justifying why previous solutions successfully described learning from small initial weights without incorporating their fine-scale structure.
Finally, we discuss the implications of these findings for continual learning, reversal learning and learning of structured knowledge.
Taken together, our results provide a mathematical toolkit for understanding the impact of prior knowledge on deep learning.
Introduction
A hallmark of human learning is our exquisite sensitivity to prior knowledge: what we already know affects how we subsequently learn [2].
For instance, having learned about the attributes of nine animals, we may learn about the tenth more quickly [3, 4, 5, 6].
In machine learning, the impact of prior knowledge on learning is evident in a range of paradigms including reversal learning [7], transfer learning [8, 9, 10, 11], continual learning [12, 13, 14], curriculum learning [15] and meta learning [16].
One form of prior knowledge in deep networks is the initial network state, which is known to strongly impact learning dynamics [17, 18, 19].
Even random initial weights of different variance can yield qualitative shifts in network behaviour between the lazy and rich regimes [20], imparting distinct inductive biases on the learning process.
More broadly, rich representations such as those obtained through pretraining provide empirically fertile inductive biases for subsequent
[Figure]
fine-tuning [21].
Yet while the importance of prior knowledge to learning is clear, our theoretical understanding remains limited, and fundamental questions remain about the implicit inductive biases of neural networks trained from structured initial weights.
A better understanding of the impact of initialisation on gradient-based learning may lead to improved pretraining schemes and illuminate pathologies like catastrophic forgetting in continual learning [22].
Here, we address this gap by deriving exact solutions to the dynamics of learning in deep linear networks as a function of network initialisation, revealing an intricate and systematic dependence.
We consider the setting depicted in Fig.
1A, where a network is trained with standard gradient descent from a potentially complex initialisation.
When trained on the same task, different initialisations can radically change the network’s learning trajectory (Fig.
1B-D).
Our approach, based on a matrix
Riccati formalism [1], provides explicit analytical expressions for the network output over time (Fig.
1B-D dotted).
While simple, deep linear networks have a non-convex loss landscape and have been shown to recapitulate several features of nonlinear deep networks while retaining mathematical tractability.
1.1 Contributions
• We derive an explicit solution for the gradient flow of the network function, internal representational similarity, and finite-width neural tangent kernel of over- and under-complete two-layer deep linear networks for a rich class of initial conditions (Section 3).
• We characterise a set of random initial network states that exhibit fast, exponential learning dynamics and yet converge to rich neural representations.
Dissociating fast and slow learning dynamics from the rich and lazy learning regimes (Section 4).
• We analyse how weights dynamically align to task-relevant structure over the course of learning, going beyond prior work that has assumed initial alignment (Section 5).
• We provide exact solutions to continual learning dynamics, reversal learning dynamics and to the dynamics of learning and revising structured representations (Section 6).
1.2 Related work
Our work builds on analyses of deep linear networks [23, 1, 17, 24, 10, 25, 26, 27], which have shown that this simple model nevertheless has intricate fixed point structure and nonlinear learning dynamics reminiscent of phenomena seen in nonlinear networks.
A variety of works has analysed convergence [28, 29], generalisation [10, 30, 31], and the implicit bias of gradient descent [32, 33, 34, 35].
These works mostly considers the tabula rasa case of small initial random weights, for which exact solutions are known [17].
By contrast our formalism describes dynamics from a much larger class of initial conditions and can describe alignment dynamics that do not occur in the tabula rasa setting.
Most directly, our results build from the matrix Riccati formulation proposed by [1].
Connecting this formulation and matrix factorisation problems yields a better characterisation of the convergence rate [27].
We extend and refine the matrix Riccati result to obtain the dynamics of over- and undercomplete networks; to obtain numerically stable forms of the matrix equations; and to more explicitly reveal the impact of initialisation.
A line of theoretical research has considered online learning dynamics in teacher-student settings [36, 37, 38], deriving ordinary differential equations for the average learning dynamics even in nonlinear networks.
However, solving these equations requires numerical integration.
By contrast, our approach provides explicit analytical solutions for the more restricted case of deep linear networks.
Other approaches for analysing deep network dynamics include the Neural Tangent Kernel (NTK) [39, 40, 41] and the mean field approach [42, 43, 44].
While the former can describe nonlinear networks but not the learning dynamics of hidden representations, the later yields a description of representation learning dynamics in wide networks in terms of a partial differential equation.
Our work is similar in seeking a subset of more tractable models that are amenable to analysis, but we focus on the impact of initialisation on representation learning dynamics and explicit solutions.
A large body of work has investigated the effect of different random initialisations on learning in deep networks.
The role of initialisation in the vanishing gradient problem and proposals for better initialisation schemes have been illuminated by several works drawing on the central limit theorem [45, 17, 46, 18, 47], reviewed in [48, 19, 49].
These approaches typically guarantee that gradients do not vanish at the start of learning, but do not analytically describe the resulting learning trajectories.
Influential work has shown that network initialisation variance mediates a transition from rich representation learning to lazy NTK dynamics [20], which we analyse in our framework.
Preliminaries and setting
Consider a supervised learning task in which input vectors x n ∈ R
N i from a set of P training pairs { ( x n , y n ) } n =
P have to be associated with their target output vectors y n ∈
R
N
o .
We learn this task with a two-layer linear network model (Fig.
1A) that produces the output prediction
[Equation]
) with weight matrices W 1 ∈ R N h
× N i and W 2 ∈ R N o × N h , where N h is the number of hidden units.
The network’s weights are optimised using full batch gradient descent with learning rate η (or respectively time constant τ = 1 / η ) on the mean squared error loss
[Equation]
where ⟨·⟩ denotes the average over the dataset.
The input and input-output correlation matrices of the dataset are
[Equation]
Finally, the gradient optimisation starts from an initialisation W 2 (0) , W 1 (0) .
Our goal is to understand the full time trajectory of the network’s output and internal representations as a function of this initialisation and the task statistics.
Our starting point is the seminal work of Fukumizu [1], which showed that the gradient flow dynamics could be written as a matrix Riccati equation with known solution.
In particular, defining
[Equation]
the continuous time dynamics of the matrix QQ T from initial state Q (0) is
[Equation]
[Equation]
if the following four assumptions hold:
Assumption 2.1 The dimensions of the input and target vectors are identical, that is N i = N o .
Assumption 2.2 The input data is whitened, that is ˜Σ xx
= I .
Assumption 2.3 The network’s weight matrices are zero-balanced at the beginning of training, that is W 1 (0) W 1 (0) T = W 2 (0) T W 2 (0) .
If this condition holds at initialisation, it will persist throughout training [17, 24].
Assumption 2.4 The input-output correlation of the task and the initial state of the network function have full rank, that is rank( ˜Σ xy )
=
rank( W 2 (0) W 1 (0))
=
N i = N o .
This implies that the network is not bottlenecked, i.e. N h ≥ min( N i , N o ) .
For completeness, we include a derivation of this solution in Appendix A.
Rather than tracking the weights’ dynamics directly, this approach tracks several key statistics collected in the matrix
[Equation]
which can be separated into four quadrants with intuitive meaning: the off-diagonal blocks contain the network function
[Equation]
while the on-diagonal blocks contain the correlation structure of the weight matrices.
These permit calculation of the temporal evolution of the network’s internal representations including the taskrelevant representational similarity matrices (RSM) [50], i.e. the kernel matrix ϕ ( x ) T ϕ ( x ′ ) , of the neural representations in the hidden layer
[Equation]
where + denotes the pseudoinverse; and the network’s finite-width neural tangent kernel [39, 40, 41]
[Equation]
where I is the identity matrix and ⊗ is the Kronecker product.
For a derivation of these quantities see
Appendix B. Hence, the solution in Equation 5 describes important aspects of network behaviour.
However, in this form, the solution has several limitations.
First, it relies on general matrix exponentials and inverses, which are a barrier to explicit understanding.
Second, when evaluated numerically, it is often unstable.
And third, the equation is only valid for equal input and output dimensions.
In the following section we address these limitations.
Implementation and simulation Simulation details are in Appendix H. Code to replicate all simulations and plots are available online 1 under a GPLv3 license and requires <6 hours to execute on a single AMD Ryzen 5950x.
Exact learning dynamics with prior knowledge
In this section we derive an exact and numerically stable solution for QQ T that better reveals the learning dynamics, convergence behaviour and generalisation properties of two-layer linear networks with prior knowledge.
Further, we alter the equations to be applicable to equal and unequal input and output dimensions, overcoming Assumption 2.1).
To place the solution in a more explicit form, we make use of the compact singular value decomposition.
Let the compact singular value decomposition of the initial network function and the input-output correlation of the task be
[Equation]
Here, U and ˜U ∈ R N o × N m denote the left singular vectors, S and ˜S ∈ R N m × N m the square matrix with ordered, non-zero eigenvalues on its diagonal and V and ˜V ∈ R N i × N m the corresponding right singular vectors.
For unequal input-output dimensions ( N i ̸ = N o ) the right and left singular vectors are therefore not generally square and orthonormal.
Accordingly, for the case N i > N o , we define ˜U ⊥ ∈ R N o × ( N o − N i ) as a matrix containing orthogonal column vectors that complete the basis, i.e., make (cid:2) ˜U ˜U ⊥ (cid:3) orthonormal.
Conversely, we define ˜V ⊥ ∈ R N i × ( N i − N o ) for the case of N i > N o .
[Figure]
C
D
Assumption 3.1 Define B = U T ˜U + V T ˜V and C =
U T ˜U − V T ˜V .
B is non-singular.
Theorem 3.1 Under the assumptions of whitened inputs, 2.2, zero-balanced weights 2.3, full rank 2.
[Equation]
and B non-singular 3.1, the temporal dynamics of QQ T are
[Equation] [Equation]
with
Z = 
[Equation]
For a proof of Theorem 3.1 please refer to Appendix C.
With this solution we can calculate the exact temporal dynamics of the loss, network function, RSMs and NTK (Fig.
2A, B).
As the solution contains only negative exponentials, it is numerically stable and provides high precision across a wide range of learning rates and network architectures (Fig.
2C, D).
We note that a solution for the weights W 1 ( t ) and W 2 ( t ) , i.e., Q ( t ) , can be derived up to a time varying orthogonal transformation as demonstrated in Appendix C. Further, as time-dependent variables only occur in matrix exponentials of diagonal matrices of negative sign, the network approaches a steady state solution.
Theorem 3.2 Under the assumptions of Theorem 3.1, the network function converges to the global minimum ˜U˜S ˜V T and acquires a rich task-specific internal representation, that is W T 1 W 1 = ˜V˜S ˜V T and W 2 W T 2 = ˜U˜S ˜U T .
The proof of Theorem 3.2 is in Appendix C. We now turn to several implications of these results.
Rich and lazy learning regimes and generalisation
Recent results have shown that large deep networks can operate in qualitatively distinct regimes that depend on their weight initialisations [20, 51], the so called rich and lazy regimes.
In the rich regime, learning dynamics can be highly nonlinear and lead to task-specific solutions thought
A
Animal
Plant
Bird Algae Fish
C
F B
D
E to lead to favourable generalisation properties [20, 25, 51].
By contrast, the lazy regime exhibits simple exponential learning dynamics and exploits high-dimensional nonlinear projections of the data produced by the initial random weights, leading to task-agnostic representations that attain zero training error but possibly lower generalisation performance [39, 40, 41].
Traditionally, the rich and lazy learning regimes have been respectively linked to low and high variance initial weights (relative to the network layer size).
To illustrate these phenomena, we consider a semantic learning task in which a set of living things have to be linked to their position in a hierarchical structure (Fig.
3A)
The representational similarity of the input of the task ( ˜V˜S ˜V T ) reveals its inherent structure (Fig.
3B).
For example, the representations of the two fishes are most similar to each other, less similar to birds and least similar to plants.
Likewise, the representational similarity of the task’s target values ( ˜U˜S ˜U T ) reveals the primary groups among which items are organised.
As a consequence, one can for example predict from an object being a fish that it is an animal and from an object being a plant that it is not a bird.
Reflecting these structural relationships in internal representations can allow the rich regime to generalise in ways the lazy regime cannot.
Crucially, QQ T ( t ) contains the temporal dynamics of the weights’ representational similarity and therefore can be used to study if a network finds a rich or lazy solution.
When training a two layer network from random small initial weights, the weights’ input and output
RSM (Fig.
3C, upper left and lower right quadrant) are identical to the task’s structure at convergence.
However, when training from large initial weights, the RSM reveals that the network has converged to a lazy solution (Fig. 3D).
We emphasise that the network function in both cases is identical (Fig.
3C, D, lower left quadrant).
And while their final loss is identical too, their learning dynamics evolve slow and step-wise in the case of small initial weights and fast and exponentially in the case of large initial weights (Fig. 3F), as predicted by previous work [20].
However, from Theorem 3.2 it directly follows that our setup is guaranteed to find a rich solution in which the weights’ RSM is identical to the task’s RSM, i.e., W T 1 W 1 =
˜ V ˜ S
˜ V T and W 2 W T 2 = ˜U˜S ˜U T .
Therefore, as zero-balanced weights may be large, there exist initial states that converge to rich solutions while evolving as rapid exponential learning curves (Fig.
3E, F).
Crucially, these initialisations are task-agnostic, in the sense that they are independent of the task structure (cf.
This finding applies to any learning task with well defined input-output correlation.
For additional simulations see
Appendix D.
Hence our equation can describe the change in dynamics from step-like to exponential with increasing weight scale, and separate this dynamical phenomenon from the structure of internal representations.
Decoupling dynamics
The learning dynamics of deep linear networks depend on the exact initial values of the synaptic weights.
Previous solutions studied learning dynamics under the assumption that initial network weights are “decoupled”, such that the initial state of the network and the task share the same singular vectors, i.e. that U = ˜U and V = ˜V [17].
Intuitively, this assumption means that there is no cross-coupling between different singular modes, such that each evolves independently.
However, this assumption is violated in most real-world scenarios.
As a consequence, most prior work has relied on the empirical observation that learning from tabula rasa small initial weights occurs in two phases:
First, the network’s input-output map rapidly decouples; then subsequently, independent singular modes are learned in this decoupled regime.
Because this decoupling process is fast when training begins from small initial weights, the learning dynamics are still approximately described by the temporal learning dynamics of the singular values assuming decoupling from the start.
This dynamic has been called a silent alignment process [26].
Here we leverage our matrix Riccati approach to analytically study the dynamics of this decoupling process.
We begin by deriving an alternate form of the exact solution that eases the analysis.
Theorem 5.1 Let the weight matrices of a two layer linear network be initialised by W
= A (0) ˜V T and W 2 = ˜UA (0) T , where A (0) ∈
R N h
× N i is an arbitrary, invertible matrix.
Then, under the assumptions of equal input-output dimensions 2.1, whitened inputs 2.2, zero-balanced weights 2.
and full rank 2.4, the temporal dynamics of QQ T are fully determined by
[Equation]
For a proof of Theorem 5.1, please refer to Appendix E. We remark that this form is less general than that in Theorem 3.1, and in particular implies UV = ˜U ˜V .
Here the matrix A T A represents the dynamics directly in the SVD basis of the task.
Off-diagonal elements represent counterproductive coupling between different singular modes (for instance, [ A T A ] 21 is the strength of connection from input singular vector 1 to output singular vector 2, which must approach zero to perform the task perfectly), while on-diagonal elements represent the coupling within the same mode (for instance, [ A T A ] 11 is the strength of connection from input singular vector 1 to output singular vector 1, which must approach the associated task singular value to perform the task perfectly).
Hence the decoupling process can be studied by examining the dynamics by which A T A becomes approximately diagonal.
The outer inverse in Equation 13 renders it difficult to study high dimensional networks analytically.
Therefore, we focus on small networks with input and output dimension N i = 2 and N o = 2 , for which a lengthy but explicit analytical solution is given in Appendix E.
In this setting, the structure of the weight initialisation and task are encoded in the matrices
[Equation]
A
B
E where the parameters a 1 (0) and a 2 (0) represent the component of the initialisation that is aligned with the task and b (0) represents cross-coupling, such that taking b (0)
=
recovers previously known and more restricted solutions for the decoupled case [17].
We use this setting to demonstrate two features of the learning dynamics.
Decoupling dynamics.
First, we track decoupling by considering the dynamics of the off-diagonal element b ( t ) (Fig.
4D-F red lines).
At convergence, the off-diagonal element shrinks to zero as shown in
Appendix E. However, strikingly, b ( t ) can exhibit non-monotonic trajectories with transient peaks or valleys partway through the learning process.
In particular, in Appendix E we derive the time of the peak magnitude as t peak = τ 4 s ln s
[Equation]
s
[Equation] [Equation]
a 2 − b
[Equation]
0) 2 (Fig.
4F green dotted line), which coincides approximately with the time at which the on-diagonal element is half learned.
If initialised from small random weights, the off-diagonal remains near-zero throughout learning, reminiscent of the silent alignment effect [26].
For large initialisations, no peak is observed and the dynamics are exponential.
At intermediate initialisations, the maximum of the off-diagonal is reached before the singular mode is fully learned (Appendix E).
Intuitively, a particular input singular vector can initially project appreciably onto the wrong output singular vector, corresponding to initial misalignment.
This is only revealed when this link is amplified, at which point corrective dynamics remove the counterproductive coupling, as schematised in Fig.
4B. We report further measurements of decoupling in
Appendix E.
Effect of initialisation variance .
Next, we revisit the impact of initialisation scale for the on-diagonal dynamics.
As shown in Fig.
4D-F, as the initialisation variance grows the learning dynamics change from sigmoidal to exponential, possibly displaying more complex behavior at intermediate variance (Appendix E).
In this simple setting we can analyse this transition in detail.
Taking s 1 = s
= s as in
Fig.
4F and | a 1 (0) | , | a 2 (0) | , | b (0) | ≪ 1 , we recover a sigmoidal trajectory,
[Equation]
(15) while for | a 1 (0) | , | a 2 (0) | , | b (0) | ≫
the dynamics of the on-diagonal element a 1 is close to exponential (Fig.
4D-F left and right columns).
We examine larger networks in Appendix E.
[Figure]
D
E
F
G
H
Applications
The solutions derived in Sections 3 and 5 provide tools to examine the impact of prior knowledge on dynamics in deep linear networks.
So far we have traced general features of the behaviour of these solutions.
In this section, we use this toolkit to develop accounts of several specific phenomena
Continual Learning Continual learning (see [12] for a review) and the pathology of catastrophic forgetting have long been a challenge for neural network models [22, 53, 54].
A variety of theoretical work has investigated aspects of continual learning [55, 56, 57, 58, 59].
In this setting, starting from an initial set of weights, a network is trained on a sequence of tasks with respective input-output correlations T 1 =
˜Σ yx 1 , T 2 = ˜Σ yx 2 , T 3 =
˜Σ yx 3 , ... .
As shown in Fig.
5A, our dynamics immediately enable exact solutions for the full continual learning process, whereby the final state after training on one task becomes the initial network state for the next task.
These solutions thus reveal the exact time course of forgetting for arbitrary sequences of tasks.
Training on later tasks can overwrite previously learned knowledge, a phenomenon known as catastrophic forgetting [22, 53, 54].
From Theorem 3.2 it follows that from any arbitrary zerobalanced initialisation 2.3, the network converges to the global optimum such that the initialisation is completely overwritten and forgetting is truly catastrophic.
In particular, the loss of any other task T i after training to convergence on task T j is L i ( T j ) = 1 / 2 || ˜Σ yxj − ˜Σ yxi || 2 F + c , where c is a constant that only depends on training data of task T
i (Appendix F).
As a consequence, the amount of forgetting, i.e. the relative change of loss, is fully determined by the similarity structure of the tasks and thus can be fully determined for a sequence of tasks before the onset of training (Fig.
5B,E,
Appendix F).
For example, the amount of catastrophic forgetting in task T a , when training on task T c after having trained the network on task T b is L a ( T c ) − L a ( T b ) .
As expected, our results depend on our linear setting and tanh or ReLU nonlinearities can show different behaviour, typically increasing the amount of forgetting (Fig.
5B,D,E).
Further, in nonlinear networks, weights become rapidly unbalanced and forgetting values that are calculated before the onset of training do not predict the actual outcome (Fig.
5B-E).
In summary, our results link exact learning dynamics with catastrophic forgetting and thus provide an analytical tool to study the mechanisms and potential counter measures underlying catastrophic forgetting.
Reversal learning During reversal learning, pre-existing knowledge has to be relearned, overcoming a previously learned relationship between inputs and outputs.
For example, reversal learning occurs when items of a class are mislabeled and later corrected.
We show analytically, that reversal learning in fact does not succeed in deep linear networks (Appendix G).
The pre-existing knowledge lies exactly on the separatrix of a saddle point causing the learning dynamics to converge to zero (Fig.
6A).
In contrast, the learning still succeeds numerically, as any noise will perturb the dynamics off the saddle point, allowing learning to proceed (Fig.
6A).
However, the dynamics still slow in the vicinity of the saddle point, providing a theoretical explanation for catastrophic slowing in deep linear networks [60].
We note that the analytical solution requires an adaptation of Theorem 3.1, as B is generally not invertible in the case of reversal learning (Appendix G).
Further, as is revealed by the exact learning dynamics (Appendix G), shallow networks do succeed without exhibiting catastrophic slowing during reversal learning (Fig. 6B).
Revising structured knowledge Knowledge is often organised within an underlying, shared structure, of which many can be learned and represented in deep linear networks [25].
For example, spatial locations can be related to each other using the same cardinal directions, or varying semantic knowledge can be organised using the same hierarchical tree.
Here, we investigate if deep linear networks benefit from shared underlying structure.
To this end, a network is first trained on the three-level hierarchical tree of Section 4 (eight items of the living kingdom, each with a set of eight associated features), and subsequently trained on a revised version of the hierarchy.
The revised task varies the relation of inputs and outputs while keeping the same underlying tree structure.
If the revision involves swapping two neighbouring nodes on any level of the hierarchy, e.g. the identity of the two fish on the lowest level of the hierarchy (Fig.
6C, top), the task is identical to reversal learning, leading to catastrophically slowed dynamics (Fig.
6E-F, top).
When training the network on a new hierarchical tree with identical items but a new set of features, like a colour hierarchy (Fig.
6C, bottom), there is no speed advantage in comparison to a random initialisation with similar initial variance (Fig.
6E-F, bottom).
Importantly, from Theorem 3.2 it follows, that the learning process can be sped up significantly by initialising from large zero-balanced weights, while converging to a global minimum with identical generalisation properties as when training from small weights (Fig.
6G-H).
In summary, having incorporated structured knowledge before revision does not speed up or even slows down learning in comparison to learning from random zero-balanced weights.
Notably, that is despite the tasks’ structure being almost identical (Fig.
3B and Fig.
6D).
Discussion
We derive exact solutions to the dynamics of learning with rich prior knowledge in a tractable model class: deep linear networks.
While our results broaden the class of two-layer linear network problems that can be described analytically, they remain limited and rely on a set of assumptions (2.1-2.4).
In particular, weakening the requirement that the input covariance be white and the weights be zero-balanced would enable analysis of the impact of initialisation on internal representations.
Nevertheless, these solutions reveal several insights into network behaviour.
We show that there exists a large set of initial values, namely zero-balanced weights 2.3, which lead to task-specific representations; and that large initialisations lead to exponential rather than sigmoidal learning curves.
We hope our results provide a mathematical toolkit that illuminates the complex impact of prior knowledge on deep learning dynamics.
Acknowledgments and Disclosure of Funding
L.B. was supported by the Woodward Scholarship awarded by Wadham College, Oxford and the
Medical Research Council [MR/N013468/1].
C.D. and A.S. were supported by the Gatsby Charitable
Foundation (GAT3755).
Further, A.S. was supported by a Sir Henry Dale Fellowship from the
Wellcome Trust and Royal Society (216386/Z/19/Z) and the Sainsbury Wellcome Centre Core Grant (219627/Z/19/Z).
A.S. is a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains program.
J.F. was supported by the Howard Hughes Medical Institute.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See section H (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[-In the case of equal input-output dimensions ˜U ⊥ = ˜V ⊥ = 
-]
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
[-= 
-]
[-As t tends to infinity lim t →∞ b ( t ) = 
-]
[-0  and equal diagonal   5 0 0 00 5 0 00 0 5 00 0 
-]
[-We characterise these dynamics considering the case where s 1 = s 
-]
[-The diagonal elements simplify in the cases where s 1 = s 
-]
[-For | a 1 (0) | , | a 2 (0) | , | b (0) | ≫ 
-]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-W 
-]
[-= 10 , N o = 
-]
[-N i = 4 , N h = 5 , N o = 
-]
[-= 10 , N o = 
-]