Exact Learning Dynamics Of Deep Linear Networks With Prior Knowledge
Anonymous Author(s)
AffiliationAddress
Abstract
Learning in deep neural networks is known to depend critically on the knowledgeembedded in the initial network weights.
However, few theoretical results haveprecisely linked prior knowledge to learning dynamics.
Here we derive exactsolutions to the dynamics of learning with rich prior knowledge in deep linearnetworks by generalising Fukumizu’s matrix Riccati solution [1].
While simple,deep linear networks retain a non-convex loss landscape and nonlinear learningdynamics that depend in detail on the initial weights of the network.
We obtainexplicit expressions for the evolving network function, hidden representationalsimilarity, and neural tangent kernel over training for a broad class of initialisationsand tasks.
We characterise a class of task-independent initialisations that radicallyalters learning dynamics from slow step-like to fast exponential trajectories whileconverging to identical representational similarity, dissociating learning trajectoriesfrom the structure of internal representations.
We discuss the implications of thisfinding for neural network weight initialisation schemes, continual learning andlearning of structured knowledge.
Finally, we characterise how network weightsdynamically align with task structure, rigorously justifying why previous solutionssuccessfully described learning from small weights without incorporating theirfine-scale structure.
Taken together, our results provide a mathematical toolkit forunderstanding the impact of prior knowledge on deep learning.
Introduction
A hallmark of human learning is our exquisite sensitivity to prior knowledge: what we already knowaffects how we subsequently learn [2].
For instance, having learned about the attributes of nineanimals, we may learn about the tenth more quickly [3, 4, 5, 6].
In machine learning, the impactof prior knowledge on learning is evident in a range of paradigms including reversal learning [7],transfer learning [8, 9, 10, 11], continual learning [12, 13, 14], curriculum learning [15] and metalearning [16].
One form of prior knowledge in deep networks is the initial network state, which isknown to strongly impact learning dynamics [17, 18, 19].
Even random initial weights of differentvariance can yield qualitative shifts in network behaviour between the lazy and rich regimes [20],imparting distinct inductive biases on the learning process.
More broadly, rich representations suchas those obtained through pretraining provide empirically fertile inductive biases for subsequentfine-tuning [21].
Yet while the importance of prior knowledge to learning is clear, our theoreticalunderstanding remains limited, and fundamental questions remain about the implicit inductive biasesof neural networks trained from structured initial weights.
A better understanding of the impact ofinitialisation on gradient-based learning may lead to improved pretraining schemes and illuminatepathologies like catastrophic forgetting in continual learning [22].
[Figure]
Here, we address this gap by deriving exact solutions to the dynamics of learning in deep linearnetworks as a function of network initialisation, revealing an intricate and systematic dependence.
We consider the setting depicted in Fig.
1A, where a network is trained with standard gradient descentfrom a potentially complex initialisation.
When trained on the same task, different initialisationscan radically change the trajectory of the network’s output (Fig.
1B-D).
Our approach, based on amatrix Riccati formalism [1], provides explicit analytical expressions for the network output overtime (Fig.
1B-D dotted).
While simple, deep linear networks have a non-convex loss landscapeand have been shown to recapitulate several features of nonlinear deep networks while retainingmathematical tractability.
1.1 Related work
Our work builds on analyses of deep linear networks [23, 1, 17, 24, 10, 25, 26], which have shownthat this simple model nevertheless has intricate fixed point structure and nonlinear learning dynamicsreminiscent of phenomena seen in nonlinear networks.
A variety of works has analysed convergence[27, 28], generalisation [10, 29, 30], and the implicit bias of gradient descent [31, 32, 33, 34].
Theseworks mostly considers the tabula rasa case of small initial random weights, for which exact solutionsare known [17].
By contrast our formalism describes dynamics from a much larger class of initialconditions and can describe alignment dynamics that do not occur in the tabula rasa setting.
Mostdirectly, our results build from the matrix Riccati formulation proposed by [1].
We extend and refinethis result to obtain the dynamics of over- and under-complete networks; to obtain numerically stableforms of the matrix equations; and to more explicitly reveal the impact of initialisation.
A line of theoretical research has considered online learning dynamics in teacher-student settings[35, 36, 37], deriving ordinary differential equations for the average learning dynamics even innonlinear networks.
However, solving these equations requires numerical integration.
By contrast,our approach provides explicit analytical solutions for the more restricted case of deep linear networks.
Other approaches for analysing deep network dynamics include the Neural Tangent Kernel (NTK)[38, 39, 40], which can describe nonlinear networks but not representation learning dynamics, andthe mean field approach [41, 42, 43], which yields a description of representation learning dynamicsin wide networks in terms of a PDE.
Our work is similar in seeking a subset of more tractable modelsthat are amenable to analysis, but we focus on the impact of initialisation on representation learningdynamics and explicit solutions.
A large body of work has investigated the effect of different random initialisations on learningin deep networks.
The role of initialisation in the vanishing gradient problem and proposals forbetter initialisation schemes have been illuminated by several works drawing on the central limittheorem [44, 17, 45, 18, 46], reviewed in [47, 19, 48].
These approaches typically guarantee that gradients do not vanish at the start of learning, but do not analytically describe the resulting learningtrajectories.
Influential work has shown that network initialisation variance mediates a transition fromrich representation learning to lazy NTK dynamics [20], which we analyse in our framework.
Preliminaries and setting
Consider a supervised learning task in which input vectors x n ∈ R
N i from a set of P training pairs{ ( x n , y n ) } n =
P have to be associated with their target output vectors y n ∈
R
N
o .
We learn thistask with a two-layer linear network model (Fig.
1A) that produces the output prediction
[Equation]
with weight matrices W 1 ∈ R N h
× N i and W 2 ∈ R N o × N h , where N h is the number of hiddenunits.
The network’s weights are optimised using full batch gradient descent with learning rate η (orrespectively time constant τ = 1 / η ) on the mean squared error loss
[Equation]
where ⟨·⟩ denotes the average over the dataset.
The input and input-output correlation matrices of thedataset are
[Equation]
Finally, the gradient optimisation starts from an initialisation W 2 (0) , W 1 (0) .
Our goal is to understand the full time trajectory of the network’s output and internal representations as a function of thisinitialisation and the task statistics.
Our starting point is the seminal work of Fukumizu [1], which showed that the gradient flow dynamicscould be written as a matrix Riccati equation with known solution.
In particular, defining
[Equation]
(cid:21) and F = (cid:20)
[Equation] the continuous time dynamics of the matrix QQ T from initial state Q (0) is
[Equation]
if the following four assumptions hold [1]:
Assumption 2.1 The dimensions of the input and target vectors are identical, that is N i = N o .
Assumption 2.2 The input data is whitened, that is ˜ Σ xx =
I .
Assumption 2.3 The network’s weight matrices are zero-balanced at the beginning of training,that is W 1 (0) W 1 (0) T = W 2 (0) T W 2 (0) .
If this condition holds at initialisation, it will persistthroughout training [17, 24].
Assumption 2.4 The input-output correlation of the task and the initial state of the network functionhave full rank, that is rank( ˜Σ xy ) =
rank( W 2 (0) W 1 (0))
=
N i = N o .
For completeness, we include a derivation of this solution in Appendix A.
Rather than tracking the weights’ dynamics directly, this approach tracks several key statisticscollected in the matrix [Equation] [Equation]
which can be separated into four quadrants with intuitive meaning: the off-diagonal blocks containthe network function [Equation] [Figure]
C D while the on-diagonal blocks contain the correlation structure of the weight matrices.
These permitcalculation of the temporal evolution of the network’s internal representations including the taskrelevant representational similarity matrices (RSM) [49] (i.e. the kernel matrix ϕ ( x ) T ϕ ( x ′ ) ) of theneural representations in the hidden layer [Equation] where + denotes the pseudoinverse and the network’s finite-width neural tangent kernel [38, 39, 40]
[Equation] where I is the identity matrix and ⊗ is the Kronecker product. For a derivation of these quantities see
Appendix B. Hence, the solution in Equation (5) describes important aspects of network behaviour.
However, in this form, the solution has several limitations.
First, it relies on general matrix exponentials and inverses, which are a barrier to explicit understanding.
Second, when evaluated numerically,it is often unstable.
And third, the equation is only valid for equal input and output dimensions.
Inthe following section we address these limitations.
Implementation and simulation All simulation details are in Appendix G. Code to replicate allsimulations and plots are in the supplementary material and available online under a GPLv3 license 1 .
Reproduction of all simulation takes <6 hours on a single AMD Ryzen 5950x.
Exact learning dynamics with prior knowledge
In this section we derive an exact and numerically stable solution for QQ T that better reveals thelearning dynamics, convergence behaviour and generalisation properties of two-layer linear networkswith prior knowledge.
First, we weaken the assumption of equal input and output dimensions 2.1:
Assumption 3.1 The input-output correlation and the initial state of the network have full rank,that is rank( ˜Σ xy )
=
rank( W 2 (0) W 1 (0))
=
min( N i , N o ) .
This implies that the network is notbottlenecked, i.e. N h ≥ min( N i , N o ) .
Next, to place the solution in a more explicit form, we make use of the compact singular valuedecomposition.
Let the compact singular value decomposition of the initial network function and theinput-output correlation of the task be
[Equation]
and
[Equation]
[Figure] Here, U and ˜U ∈ R N o × N m denote the left singular vectors, S and ˜S ∈ R N m × N m the square matrixwith ordered, non-zero eigenvalues on its diagonal and V and ˜V ∈ R N i × N m the corresponding rightsingular vectors. For unequal input-output dimensions ( N i ̸ = N o ) the right and left singular vectorsare therefore not generally square and orthonormal.
Accordingly, for the case N i > N o , we define˜U ⊥ ∈ R N o × ( N o − N i ) as a matrix containing orthogonal column vectors that complete the make (cid:2) ˜U ˜U ⊥ (cid:3) orthonormal.
Conversely, we define ˜V ⊥ ∈ R N i × ( N i − N o ) for the case of N i > N
Assumption 3.2 Define B = U T ˜U + V T ˜V and C =
U T ˜U − V T ˜V .
B is non-singular. Theorem 3.1 Under the assumptions of whitened inputs, 2.2, zero-balanced weights 2.3, full rank3.1, and B non-singular 3.2, the temporal dynamics of QQ T are [Equation]
[Equation]
with
Z =
[Equation]
For a proof of Theorem 3.1 please refer to Appendix C.
With this solution we can calculate the exact temporal dynamics of the loss, network function, RSMsand NTK, as we verify in Fig.
2A, B. Because it contains only negative exponentials, it is numericallystable and provides high precision across a wide range of learning rates and network architectures(Fig.
2C, D).
We note that a solution for the weights W 1 ( t ) and W 2 ( t ) , i.e., Q ( t ) , can be derived up to a timevarying orthogonal transformation as demonstrated in Appendix C. Further, as time-dependentvariables only occur in matrix exponentials of diagonal matrices of negative sign, the networkapproaches a steady state solution.
Theorem 3.2 Under the assumptions of Theorem 3.1, the network function converges to the globalminimum ˜U˜S ˜V T and acquires a rich task-specific internal representation, that is W T 1 W 1 = ˜V˜S ˜V Tand W 2 W T 2 = ˜U˜S ˜U T .
The proof of Theorem 3.2 is in Appendix C. We now turn to several implications of these results.
Rich and lazy learning regimes and generalisation
Recent results have shown that large deep networks can operate in qualitatively distinct regimesdepending on their weight initialisations [20, 50], the so called rich and lazy regimes. In the rich regime, learning dynamics can be highly nonlinear and lead to task-specific solutions thought tolead to favourable generalisation properties [20, 25, 50].
By contrast, the lazy regime exhibitssimple exponential learning dynamics and exploits high-dimensional nonlinear projections of thedata produced by the initial random weights, leading to task-agnostic representations that attain zerotraining error but possibly lower generalisation performance [38, 39, 40].
Traditionally, the richand lazy learning regimes have been respectively linked to low and high variance initial weights (inrelation to the network layer size).
To illustrate these phenomena, we consider a semantic learning task in which a set of living things haveto be correctly linked to their position in a hierarchical structure (Fig. 3A)
The representationalsimilarity of the input of the task ( ˜V˜S ˜V T ) reveals its inherent structure (Fig. 3A).
For example, therepresentations of the two fishes are most similar to each other, less similar to birds and least similarto plants.
Likewise, the representational similarity of the task’s target values ( ˜U˜S ˜U T ) reveals theprimary groups among which items are organised.
As a consequence, one can for example predictfrom an object being a fish that it is an animal and from an object being a plant that it is not abird.
Reflecting these structural relationships in internal representations can allow the rich regime togeneralise in ways the lazy regime cannot.
Crucially, QQ T ( t ) contains the temporal dynamics of theweights’ representational similarity and therefore can be used to study if a network finds a rich orlazy solution.
When training a two layer network from random small initial weights, the weights’ input and output
RSM (Fig.
3B, upper left and lower right quadrant) are identical to the task’s structure at convergence.
However, when training from large initial weights, the RSM reveals that the network has converged toa lazy solution (Fig. 3C).
We emphasise that the network function in both cases is identical (Fig.
3B, C,lower left quadrant).
And while their final loss is identical too, their learning dynamics evolve slowand step-wise in the case of small initial weights and fast and exponentially in the case of large initialweights (Fig. 3E), as predicted by previous work [20].
However, from Theorem 3.2 it directly follows that our setup, under the assumption of zero-balancedinitial weights 2.3, is guaranteed to find a rich solution in which the weights’ RSM is identicalto the task’s RSM, i.e., W T 1 W 1 = ˜V˜S ˜V T and W 2 W T 2 = ˜V˜S ˜V T .
Therefore, as zero-balancedweights may be large, there exist initial states that converge to rich solutions while evolving as rapidexponential learning curves (Fig.
3D, E).
Crucially, these initialisations are task-agnostic, in the sensethat they are independent of the task structure (cf.
This finding applies to any learning taskwith well defined input-output correlation.
For additional simulations see Appendix D. Hence ourequation can describe the change in dynamics from step-like to exponential with increasing weightscale, and separate this dynamical phenomenon from the structure of internal representations.
Exact alignment dynamics
The learning dynamics of deep linear networks depend on the exact initial values of the synapticweights.
Previous solutions [17] for the exact temporal learning dynamics relied on full alignmentbetween initial state and the target task, i.e. that U =
˜
U and V = ˜ V .
In other words, the initialstate of the network was assumed to contain part of the task’s structure before the onset of training.
However, this assumption is violated in most real-world scenarios.
As a consequence, most previouswork has relied on the empirical observation that learning from tabula rasa small initial weightsoccurs in two phases: First, the network’s singular vectors rapidly align with the task’s singular vectorsand then subsequently, the singular values are learned.
Because this alignment process is fast, whentraining begins from small initial weights, the learning dynamics are still approximately describedby the temporal learning dynamics of the singular values.
This dynamic has been called a “silentalignment” process [26].
Here we leverage our matrix Riccati approach to provide exact solutionsto the dynamics of this alignment process.
Further, we lift the assumption of small random weightsand characterise the transition from stage-like to exponential learning trajectories as initialisationvariance increases, as seen in Section 4.
We begin by deriving an alternate form that eases analysis ofalignment in particular.
Theorem 5.1 Assume that N i
=
N o = N h .
Let the weight matrices of a two layer linear networkbe initialised by W
= A (0) ˜V T and W 2 = ˜UA (0) T , where A (0) ∈
R N h
× N i is an arbitrary,invertible matrix.
Then, under the assumptions of equal input-output dimensions 2.1, whitened inputs
2.2, zero-balanced weights 2.3 and full rank 2.4, the temporal dynamics of QQ T are fully determinedby
[Equation]
[Equation]
We say that a network function is aligned with a task if the network’s and the task’s left and rightsingular vectors (eq. 10) are identical, that is, ˜U = U ( t ) and ˜V = V ( t ) .
Theorem 5.2 The network function is aligned if and only if the off-diagonal values of A T A ( t ) arezero.
For proofs of Theorems 5.1 and 5.2, please refer to Appendix E. We remark that the form of theinitial weights is less general than that in Theorem 3.1, and in particular implies UV = ˜U ˜V .
Asshown in Fig.
4A, using eq.
13 we can exactly describe the temporal dynamics of the on- andoff-diagonal values of A T A ( t ) .
Intuitively, off-diagonal values in A T A represent counterproductivecross-coupling between different singular vector modes, and eventually converge to zero despitetransient peaks.
From Theorem 5.2 it follows that the temporal dynamics of the alignment processdepend on the trajectories of the off-diagonal entries in A T A .
However, the outer inverse in eq. 13 renders it difficult to study high dimensional networks analytically.
Therefore, we first focus on small networks with input and output dimension N i = 2 and N o = 2 , forwhich a lengthy but explicit analytical solution is given in Appendix E.
In this setting, the structure ofthe weight initialisation and task are encoded in the matrices
[Equation]
and ˜S =
[Equation]
where the parameters a 1 (0) and a 2 (0) represent the component of the initialisation that is alignedwith the task, and b (0) represents cross-coupling, such that taking b (0)
=
recovers previouslyknown and more restricted solutions for the aligned case [17].
We use this setting to demonstrate twofeatures of the learning dynamics.
Alignment dynamics.
First, we track alignment by considering the dynamics of the off-diagonalelement b ( t ) (Fig.
4D-F red lines).
At convergence, the off-diagonal element shrinks to zero asshown in Appendix E. However, strikingly, it can exhibit non-monotonic trajectories with transient
[Figure]
peaks or valleys partway through the learning process.
In particular, in Appendix E we derivethe time of the peak magnitude as t peak = τ 4 s ln s ( s − a 1 − a 2 ) a 1 a 2 − b (0) 2 (Fig.
4F green dotted line), whichcoincides approximately with the time at which the on-diagonal element is half learned.
Fromsmall random weights, the off-diagonal remains near-zero throughout learning, reminiscent of thesilent alignment effect [26].
For large initialisations, no peak is observed and the dynamics areexponential.
At intermediate initialisations, the maximum of the off-diagonal is reached before thesingular mode is fully learned (Appendix E).
Intuitively, a particular input singular vector can initiallyproject appreciably onto the wrong output singular vector, corresponding to initial misalignment,but this is only revealed when this link is amplified, at which point corrective dynamics remove thecounter-productive coupling, as schematised in Fig.
4B.
In principle, all the matrice element arecoupled, however as shown in Fig.
4 the alignment dynamics have a minimal effect on the on-diagonaltrajectories.
We report further measurements of alignment in Appendix E.
Effect of initialisation variance .
Next, we revisit the impact of initialisation scale for the on-diagonaldynamics.
As shown in Fig. 4D-F, as the initialisation variance grows the learning dynamics changefrom sigmoidal to exponential, possibly displaying more complex behavior at intermediate variance(Appendix E).
In this simple setting we can analyse this transition in detail.
Taking s 1 = s
= s as in Fig. 4F and | a 1 (0) | , | a 2 (0) | , | b (0) | ≪ 1 , we recover a sigmoidal trajectory, [Equation]
[Equation]
while for | a 1 (0) | , | a 2 (0) | , | b (0) | ≫
the dynamics of the on-diagonal element a 1 is close to exponential (Fig.
4D-F left and right columns).
We examine larger networks in Appendix E.
Applications
The solutions derived in Sections 3 and 5 provide tools to examine the impact of prior knowledge ondynamics in deep linear networks.
So far we have traced general features of the behaviour of thesesolutions.
In this section, we use this toolkit to develop accounts of several specific phenomena.
Continual Learning Continual learning (see [12] for a review) and the pathology of catastrophicforgetting have long been a challenge for neural network models [22, 52, 53].
A variety of theoreticalwork has investigated aspects of continual learning [54, 55, 56, 57, 58].
In this setting, starting froman initial set of weights, a network is trained on a sequence of tasks with respective input-output correlations T 1 =
˜Σ yx 1 , T 2 = ˜Σ yx 2 , T 3 =
˜Σ yx 3 , ... .
As shown in Fig.
5A, our dynamics immediatelyenable exact solutions for the full continual learning process, whereby the final state after training onone task becomes the initialisation for the next task.
These solutions thus reveal the exact time courseof forgetting for arbitrary sequences of tasks.
Training on later tasks can overwrite previously learned knowledge, a phenomenon known ascatastrophic forgetting [22, 52, 53].
From Theorem 3.2 it follows that from any arbitrary zerobalanced initialisation 2.3, the network converges to the global optimum such that the initialisationis completely overwritten and forgetting is truly catastrophic.
In particular, the loss of any othertask j after training to convergence on task i is L ( T j )
= 1 / 2 || ˜Σ yxj − ˜Σ yxi || 2 F .
Hence the amount offorgetting after convergence is fully determined by the similarity structure of the two tasks (Fig.
5B),and we can fully predict catastrophic forgetting for a sequence of tasks of arbitrary length beforetraining onset (Fig.
5D).
As expected, our results depend on our linear setting and tanh or ReLUnonlinearities can show different behaviour, typically increasing the amount of forgetting (Fig.
5B,D).
Further, in nonlinear networks, weights become rapidly unbalanced and forgetting values that arecalculated before the onset of training do not predict the actual outcome (Fig.
5C,D).
In summary,our results link exact learning dynamics with catastrophic forgetting and thus provide an analyticaltool to study the mechanisms and potential counter measures underlying catastrophic forgetting.
[Figure]
converges Reversal learning We study knowledge revision, that is, how preexisting knowledge can be altered.
We consider the setting in whicha network first learns about a hierarchical dataset [17] (e.g., the livingkingdom, Fig.
3A), and subsequently must alter this knowledge.
Weexamine reversal learning, which requires exchanging knowledgeabout two items, for instance as might arise when items are mislabeled and later corrected.
We show analytically and numericallyin Appendix F that reversal learning in fact does not succeed indeep linear networks violating the assumption that B is invertible3.2 .The first task lies exactly on the separatrix of a saddle point,and the dynamics converge to zero (Fig.
6A), though any noise willallow learning to proceed.
However, the dynamics still slow in thevicinity of the saddle point, providing a theoretical explanation forcatastrophic slowing in deep linear networks [59].
By contrast, shallow networks do not exhibit catastrophic slowing during reversallearning (Fig. 6B), as we show in Appendix F.
Revising structured knowledge Next, we consider learning adifferent hierarchy for the same set of items, for instance learning acolour hierarchy (Appendix F) after the living hierarchy [17].
Thesetasks share structure at a highly abstract level, and differ purely intheir singular vectors, not their singular values.
We numerically andanalytically show that deep linear networks are unable to exploit thissimilarity, and the network undergoes similar learning dynamics tothose for random weights of similar scale (Appendix F).
In otherwords, deep linear networks in our setting do not have an advantagein learning subsequent abstract structures of a similar form.
Discussion
We derive exact solutions to the dynamics of learning with rich prior knowledge in a tractablemodel class: deep linear networks.
While our results broaden the class of two-layer linear networkproblems that can be described analytically, they remain limited and rely on a set of assumptions(2.1-3.1).
In particular, weakening the requirement that the input covariance be white and the weightsbe zero-balanced would enable analysis of the impact of initialisation on internal representations.
Nevertheless, these solutions reveal several insights into network behaviour.
We show that thereexists a large set of initial values, namely zero-balanced weights 2.3, which lead to task-specificrepresentations; and that large initialisations lead to exponential rather than sigmoidal learning curves.
We hope our results provide a mathematical toolkit that illuminates the complex impact of priorknowledge on deep learning dynamics. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
{+[Yes] See section 
+}
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See section G
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See section
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]