No Free Lunch from Deep Learning in Neuroscience: A Case Study through Models of the Entorhinal-Hippocampal Circuit
Abstract
Research in Neuroscience, as in many scientific disciplines, is undergoing a renaissance based on deep learning.
Unique to Neuroscience, deep learning models canbe used not only as a tool but interpreted as models of the brain.
The central claimsof recent deep learning-based models of brain circuits are that they make novelpredictions about neural phenomena or shed light on the fundamental functionsbeing optimized.
We show, through the case-study of grid cells in the entorhinalhippocampal circuit, that one may get neither.
We begin by reviewing the principlesof grid cell mechanism and function obtained from first-principles modeling efforts,then rigorously examine the claims of deep learning models of grid cells.
Usinglarge-scale hyperparameter sweeps and theory-driven experimentation, we demonstrate that the results of such models may be more strongly driven by particular,non-fundamental, and post-hoc implementation choices than fundamental truthsabout neural circuits or the loss function(s) they might optimize.
We discuss whythese models cannot be expected to produce accurate models of the brain withoutthe addition of substantial amounts of inductive bias, an informal No Free Lunchresult for Neuroscience.
Based on first principles work, we provide hypotheses forwhat additional loss functions will produce grid cells more robustly.
In conclusion,caution and consideration, together with biological knowledge, are warranted inbuilding and interpreting deep learning models in Neuroscience.
Introduction
Over the past decade, deep learning (DL) has underpinned nearly every success story in machinelearning, e.g., [44, 4] and increasingly many advances in fundamental science research, e.g., [26].
In neuroscience, deep learning is similarly gaining widespread adoption as a useful method forbehavioral and neural data analysis [41, 39, 21, 33, 30, 36].
But DL offers a unique contribution to neuroscience that goes beyond its role in other fields, in thatdeep networks can be viewed as models of the brain.
The success of DL in matching or surpassinghuman performance means it is now possible to construct models of circuits that may underlie humanintelligence.
As a recent review wrote, “researchers are excited by the possibility that deep neuralnetworks may offer theories of perception, cognition and action for biological brains.
This approachhas the potential to radically reshape our approach to understanding neural systems" [42].
Further,
DL is a democratizing force for building neural circuit models of complex function.
The essential claims (and promises) of DL-based models of the brain are that 1) Because the modelsare trained on a specific optimization problem, if the resulting representations match what has beenobserved in the brain, then they reveal which optimization problem(s) the brain is solving, or 2) Thesemodels, when trained on sensible optimization problems, should generate novel predictions about thebrain’s representations and behavior.
However, given the nascent nature of such approaches and the excitement accompanying someclaims, we should examine them carefully.
In DL and deep reinforcement learning, some successesattributed to novel algorithms have been shown to instead stem from seemingly minor or unstatedimplementation choices [52, 15, 25].
In this paper, we ask whether Neuroscientists should similarlybe cautious that DL-based models of neural circuits that make specific claims about revealing thebrain’s optimizaton functions or that predict specific neural tuning curves may tell us less aboutfundamental scientific truths and more about programmers’ particular implementation choices, andmight as a result be more post hoc than predictive.
To explore these questions and point toward a systematic and circumspect use of DL for Neuroscience,we evaluate recent DL-based models of grid cells in the entorhinal-hippocampal circuit.
The medialentorhinal cortex (MEC) and hippocampus (HPC) are part of the hippocampal formation, a criticalbrain structure for learning and memory.
In a pair of Nobel-prize winning discoveries, HPC wasshown to contain place cells [38] and MEC, its cortical input, was shown to contain grid cells [22].
Place cells fire at one or several seemingly random locations in small and large environments [40],respectively, while grid cells fire in a spatially periodic pattern in all two-dimensional environments,whenever the animal is at the vertices of a hexagonal lattice [22].
Over five decades, the hippocampalformation has been central to understanding how the brain organizes spatial and episodic memory,for experimentalists and theorists alike, with many mysteries remaining.
A recent series of DLbased models of the circuit [12, 2, 46, 54, 37]) present a story that training neural circuits on pathintegration (PI) (i.e., the task of estimating one’s spatial position in an environment by integratingvelocity estimates) results in the emergence of grid cells.
We use public code from prior publications to demonstrate these results are due to implementationdetails that tell us more about those choices than they do about MEC.
By leveraging theoreticallyguided large-scale hyperparameter exploration and hypothesis-driven experimentation, we show:
Networks trained on path integration tasks almost always learn to optimally encode position,but almost never learn grid-like representations to do so.
The emergence of grid-like representations depends wholly on a specifically chosen encoding
of the supervised target, not on the task itself
, and these choices may be unrealistic.
The chosen encoding requires many other sensitive hyperparameter choices, such that small
alterations result in loss of grid-like representations.
Grid periods and period ratios depend on hyperparameter choices and are not set by the task.
Multiple grid modules, a fundamental characteristic of the grid cell system, do not emerge.
DL produces grid cells and some attendant properties only after making many specific design choicesand searching hyperparameter space to obtain such representations, effectively baking grid cells intothe task-trained networks.
It is highly improbable that DL models of path integration would haveproduced grid cells as a novel prediction from task-training, had grid cells not already beenknown to exist.
Moreover, it is unclear what interpretability or understanding these models contribute,beyond or even up to what has already been shown for these particular circuits.
These results challengethe notion that deep networks offer a free lunch for Neuroscience in terms of discovering the brain’soptimization problems or generating novel a priori predictions about single-neuron representations,and warn that caution is needed when building and interpreting such models.
Our work benefited greatly from previous publications that published their code, for which the authorsshould be commended.
By building on their code, we have been able to present novel insights that
[Figure]
b we hope will contribute to a clearer understanding of the potential risks and rewards of using andinterpreting DL models in Neuroscience.
To facilitate further research, we similarly release code 1 .
Background: grid cells
Grid cells [22] are found in the medial entorhinal cortex of mammals and are tuned to represent thespatial location of the animal as it traverses 2D space.
Each cell fires at every vertex of a triangularlattice that tiles the explored space, regardless of the speed and direction of movement throughthe space.
As a population, grid cells exhibit several striking properties that provide support for aspecialized and modular circuit.
Grid cells form discrete modules (clusters), such that all cells withina module share a common period and orientation, while different modules express discretly differentspatial periods [49].
The period ratios of successive modules have values in the range of 1.2-1.5.
The mechanism underlying grid cells is widely supported to be through attractor dynamics:
Translation-invariant lateral connectivity within the grid cell network results in a linear Turinginstability and pattern formation [7, 17, 5].
These models explain how grid cells can convert velocityinputs into updated spatial estimates, and make several predictions that have been confirmed in experiments, including most centrally the stability of low-dimensional cell-cell relationships regardless ofenvironment and behavioral state, that define a toroidal attractor dynamics [18, 55, 51, 20, 19].
Experimental approach
Path integration (PI) is the task of using self-velocity estimates to track one’s spatial position, acrucial component of spatial navigation.
The central message of existing DL models of grid cells isthat training ANNs to path integrate causes the networks to learn grid cells [12, 2, 46, 54, 37].
We follow the setup used by many previous papers: a 2.2m x 2.2m arena is created, then, spatialtrajectories (i.e. sequences of positions and velocities) are sampled.
Networks receive as inputsthe initial position and velocities, and are trained to output (a possible encoding of) the positions(Fig.
1ab) in a supervised manner.
There are multiple possible encodings of position, and as we willshow, this choice is critical.
Two simple encodings are Cartesian [12] or polar [1].
Another encodingscheme is via bump functions in 2D space, with each output assigned a single different position thattiles the space and all outputs assigned identical tuning curves [2, 46, 46, 37].
This encoding has beenequated with place cells, even though place fields tend to be heterogeneous in size and shape [40, 13],as well as in number [40], and unlike the choice of a difference-of-Gaussians (DoG) readout tuning in
ANN models, do not exhibit any inhibitory surround.
See Appendix A for position encoding details.
For all encodings, supervised learning is used to train the network via backpropagation through time.
[Figure]
Spatial tuning assessments The spatial tuning of hidden units in the networks, visualized as ratemaps,is the primary method for comparison with the brain’s grid cells.
To compute ratemaps, a trainednetwork is evaluated on long trajectories that cover the 2D environment, then each hidden unit’sactivity is binned against the true position in the environment, and counts per bin are normalized bythe number of visits to that bin.
Ratemaps of units are compared with grid cells through the gridnessscore.
We are extremely lenient with classifying a training run with particular hyperparametersa success: if even a *single* hidden unit has a grid score above a certain threshold, we say themodel possesses grid cells.
The grid score is not perfect since cells classified by grid scores representonly an upper bound on the total number of grid cells; for details, see Appendices B and C.
Networks trained on path integration tasks learn to estimate position, but
rarely learn grid cells
As [46] wrote in their section titled “Optimally encoding position yields diverse grid-like patterns,”“Why do these diverse architectures, across diverse tasks (both navigation and autoencoding), allconverge to a grid-like solution, and what governs the lattice structure of this solution?”
We demonstrate, in contrast, that most networks do not converge to a grid-like solution, instead requiring veryspecific readout tuning functions and hyperparameter choices.
Grid-like representation emergenceis determined by choices made by the programmer that, rather than relating to the path integrationobjective or biologically realistic place cells, is designed post hoc to produce grid cells.
We ran large-scale hyperparameter sweeps across common implementation choices: 1) Architectures:
RNN [14]; LSTM [24]; GRU [10]; UGRNN [11]; 2) Activation: Sigmoid; Tanh; ReLU; Linear;3)
Optimizers: SGD, Adam [31]; RMSProp [23] 4) Supervised Targets: Cartesian; Polar; highdimensional bump-like readout population code with Gaussian [2], Difference-of-Softmaxes (DoS)[46, 47, 37] or Difference-of-Gaussians (DoG) tuning curves.
5) Loss: mean squared error on theagent’s
Cartesian position [27, 12]; geodesic distance on the agent’s polar position [1]; cross entropyon a high-dimensional population of bump-like readout units [2, 46, 37] 6) Miscellaneous: recurrent& readout dropout, initialization, regularization, seed.
[Figure]
For networks with bump-like population readouts, we additionally swept: 1) Field width σ
standard deviation of the Gaussian tuning curve (often denoted σ E def =
σ in the literature); 2) Whetherthe bump-like readouts have homogeneous or heterogeneous field widths; 3)
In the case of readoutswith DoG or DoS tuning curves, the surround scale s i.e. the ratio between the inhibitory and excitatory
Gaussian’s standard deviations ( s def = σ I /σ E ); 4) Number of fields per readout unit.
Evaluating the entire hyperparameter volume is computationally prohibitive, so we evaluated a subvolume mostconsistent with previous papers, focusing on perturbations from hyperparameters that did producegrid cells.
In this sense, our search was biased toward configurations shown to produce grid cellemergence and thus our findings about the fragility of these solutions conservatively favoredthese solutions as much as possible.
All sweeps are provided in Appendix E.
To evaluate whether a network learns to optimally encode its position, we measured its positiondecoding error using previous papers’ method [2, 46, 37]: we computed position decoding error usingthe network’s output Cartesian positions (if trained on Cartesian targets) or by decoding position fromthe network’s outputs.
Any network with error < 10 cm was considered to have achieved optimalposition encoding; this threshold was chosen based on noise inherent in the decoding algorithm.
In total, we trained > 5500 networks and found that most hyperparameter configurations succeed inlearning to path integrate (Fig.
2a, Top), but few learn grid cell representations (Fig.
2a, Bottom).
This is consistent with earlier work [27, 1] demonstrating that networks can learn to path integrateand solve other navigational problems (e.g. estimating which of several environments correspond tothe current location) without lattice cells emerging as a solution.
Lattice cell emergence requires a highly specific choice of supervised target
encoding
We next sought to characterize when grid cells are learnt under different encodings of the supervisedtarget i.e., 2D position.
We tested multiple encodings: i) Cartesian, ii) Polar, iii) Gaussian, iv) a veryspecifically selected, Difference-of-Gaussian (DoG/DoS) readout and more.
We found that grid cellsdo not emerge from Cartesian, Polar or Gaussian encodings (Fig. 2abc), consistent with earlier work[27].
Only by choosing a DoG readout tuning curve did grid cells sometimes emerge.
Grid periods are parameter-dependent and multiple modules do not emerge
Next, a prominent feature of grid cells that is critical for their unambiguous encoding of positionover large scales is the existence of a discrete set of grid periodicities, which tend to scale by a roughfactor of 1 . 4 between adjacent scales [49].
We asked whether ANN models generate multiple periodsand when they did so, what hyperparameters and other choices the formed periods depended on.
[Figure]
To ensure we would obtain at least some grid cells, we fixed the readouts to be DoSs, and swept overdifferent scales of the DoS.
We found that almost all runs had a unimodal distribution of grid periods(Fig. 3a), meaning the networks learnt only one module of grid cells.
Further, we found that the period of the formed grid-like representation is completely determined bythe scale σ E of the externally imposed readout DoG (Fig. 3).
The period of the grid-like responses inevery run increased monotonically with the width of the DoG readout (Fig. 3b).
Since the models didnot result in multiple modules in a single network, we used the somewhat discrete distribution ofpeaks of the single module formed when sweeping the DoG parameter more continuously to computegrid period ratios.
These period ratios from adjacent peaks led to non-biological values (Fig. 3c).
Fourier analysis of Turing instability explains the preceding empirical results Why do only
Difference-of-Softmaxes (DoS) readouts produce grid cells?
We restate the essence of previousanalyses of first-principles models [7, 29] here, and explain why they might explain our empiricalfindings.
In a recurrent network with dynamics ˙ r ( x ) = − r ( x ) + g ( W ⋆ r ) , where x designates the neuralindex (in a continuum approximation for neurons), W ⋆ r designates the total (integrated) inputs fromthe network to the neuron at index x , and g is the non-linearity, if the recurrent weight interaction istranslationally invariant, then W ( x, x ′ )
=
W ( x − x ′ )
=
W (∆ x ) .
Under DoG interactions:
[Equation]
where ∆ x refers to the difference of indices between the neural pair linked by the weights.
The 186 evolution of activity can be decomposed into the growth and decay of Fourier components of the ratevector, which is fully determined by the Fourier transform of W , which is given by:
[Equation]
Here α E ( α I ) denotes the strength and σ E ( σ I ) denotes the scale of excitation (inhibition).
Forlinearized dynamics that approximate ˙ r
( x ) ∼ − r ( x ) + f (∆ x ) ⋆ r (i.e. g has been linearized),
[Figure]
the solution will be periodic if the maxima of ˜ f
( k ) , given by [ k ∗ ] 2 =
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
,contains sufficient power and if k ∗ ̸ = 0 .
Specifically, the condition for pattern formation is ˜ f
( k ∗ ) > 1[8, 29].
In particular, the inhibitory surround contained in f (∆ x ) , with strength σ I , is key to patternformation; if σ I → ∞ or α I → 0 , the maximum is at the origin ( k ∗ = 0 ), causing no patternformation.
Therefore, a Gaussian interaction cannot produce periodic patterns.
The connection of this theory to ANNs trained with supervised readout target P was made in [46]through the observation that dynamics with an MSE loss ||
P − W readout r || 2 can be approximatedas ˙ r
= − λr + Σ r , where Σ = PP T is the readout correlation matrix and λ is a regularizationparameter.
This dynamics now looks identical to the first dynamics with recurrent interaction matrix
W ( x, x ′ ) =
Σ xx ′ .
For identical, unimodal target readout functions with DoG tuning curves, thisreadout correlation matrix is also a difference of Gaussians (Appendix D).
By the same theoryoutlined above, Gaussian tuning curves should not produce periodic patterns.
Unmentioned implementation details matter We discovered a seemingly minor implementationdetail in several preceding papers [46, 47, 37] that proved important for the emergence of grid cells,that (to the best of our knowledge) is unmentioned in the main texts and supplements.
While thesepapers nominally refer to a Difference-of-Gaussian (DoG) readout target function (Eqn. 1), their codeinstead uses a Difference-of-Softmaxes (DoS) target function.
When we trained ideal grid-forming
ReLU networks on place-cell-like target encodings with DoG tuning curves, sweeping the receptivefield σ and surround scale s , DoG readouts did not result in grid cells (Fig. 4b), while DoS readoutsdid.
The preceding Fourier analysis shows why DoS tuning, though still more unrealistic as a placecell model than DoG, matters (Fig. 4c).
Grid cells disappear with more biologically-realistic readout functions
Place cells, to which grid cells project, differ significantly from the highly simplified single-scale,single-field ANN readouts.
Place cells have heterogeneously distributed field widths, many withmultiple fields [40, 13] (and are not DoS-like functions).
This naturally leads to the question: Willreadouts targets with multiple scales and multiple fields per place cell still produce grid cells?
We found that networks trained with multiple-field multiple-scale bump encodings per readout unitachieve comparably low position decoding error as good single-field single-scale encodings (Fig.
5a),but do not learn grid cells (Fig. 5bc).
This further demonstrates that accurate position encoding andintegration does not require grid representations.
Why path-integrating ANNs might achieve high predictivity of MEC data
We conclude by introducing a puzzle.
A recent NeurIPS spotlight [37] notes that networks trainedon single-field single-scale DoS readout encodings explain variance in mouse MEC neural activityat nearly 100% of variance explained by other mice.
In contrast, our results demonstrate that thesenetworks learn few grid cells and produce unimodal grid period distributions inconsistent withbiological grid cells, and require readout encodings inconsistent with biological place cells to do so.
How are these networks able to predict mouse MEC neural activity so well?
The neural data is not public and the analysis code is not open source, so we are unable to investigatethis puzzle.
However, we offer a conjecture with preliminary evidence.
The analysis of [37] linearlyregressed rate maps from one agent (mouse or network) onto rate maps from another mouse, and used Pearson correlation as a measure of “neural predictivity.”
We conjecture that different architectureactivation pairs achieve different neural predictivity scores because different pairs learn differentintrinsic dimensionalities that then provide richer/ poorer bases for linear regressions. predictivity from
To explore our conjecture, we trained [37]’s 5 architectures (RNN, LSTM, GRU, UGRNN, SRNN
with ReLU activations, with 5 seeds per architectureactivation pair.
For each trained network, we extracted rate maps and computed a standard linearmeasure of dimensionality called participation ratio[32], then plotted each pair’s average participationratio against the published neural predictivity.
Wefound networks with higher dimensional rate mapshave higher neural predictivity (Fig. 6).
We caution this correlation between dimensionalityand neural predictivity is not strong evidence, but weare unable to investigate further without data.
If ourconjecture is correct, it raises an important researchquestion of whether linear regression-based comparisons of ANNs with neural data might producebetter matches to biology more because ANNs have higher dimensional representations than competitors, rather than because of a detailed similarity.
We hope this section will be a call for the communityto investigate more closely comparison methods between artificial and biological networks.
Discussion
For research that uses deep networks as models of the brain, there is a fundamental obstacle to makingthe claim that a given optimization problem is what the brain is solving: If we know the responses ofa significant fraction of units from biological networks performing a certain task, we cannot inferthe loss function that the brain is optimizing since in principle, numerous different loss functionscan have the same/similar minima (Fig. 7 top).
In other words, there is typically a many-to-onemapping between loss functions and some point in state space where the functions have a minimum.
Some of the different grid models from deep learning and first principles show that this is possible[12, 2, 46, 54, 7].
Conversely, given a reasonable optimization problem that we select based onan organism’s ecological niche, we cannot infer a single solution (and thus build truly predictivesingle-cell tuning models), since there exist several minima to that optimization problem (Fig. 7bottom).
In other words, there is typically a one-to-many mapping from a loss function to its set ofsolutions.
To break this degeneracy of multiple minima and arrive at truly predictive models or abetter understanding of the brain’s optimization problems, we must acknowledge, understand, andmodel the specific inductive biases and constraints present in the biological system we are trying tomodel.
It is untenable to expect success without doing so.
This is what we refer to as an informalneural ’no-free-lunch’.
In these cases, these constraints and biases are as informative as other designprinciples, such as the loss function.
What can we learn from DL logical inductive biases?
Population-wide lowdimensional latent representations and dynamics that arise as necessary for solving difficultproblems are possibly sufficiently robust and abstracted to be predictive of population dynamicsin a neural circuit without addition of detailed biological constraints.
This can explain successesof the population-level analyses of the visualpathway [28, 3], as well as the population-levellow-dimensional dynamics of circuits solvinginference tasks [50, 43, 1, 53].
In these cases,the emergent solutions are fundamental featuresof any systems that solve the task, and by construction need not be specific to brains.
Returning to grid cells, we have conclusivelyshown that they do not generically arise in networks trained to path-integrate.
This raises thequestion of what different additional architectural, hyperparameter, and constraint choiceshad to be made in previous papers [2, 46, 47] toobtain grid-like tuning, given that they used a path integration objective with Gaussian place celltargets, requirements that are reasonable but not sufficient to obtain grid-like tuning.
Theoretical work on grid cell representations [16, 48, 35] suggests that the following two factorsare important features of the grid cell code: a very large coding range and the related property ofrobustness/intrinsic error correcting coding.
Adding these properties to the loss are likely to be amore principled way to obtain grid-like fields rather than by hand-designing a biologically unrealistic
DoG or DoS readout.
Consistent with this, the very large amount of dropout or other noise requiredin [2, 12] suggests that the coding-theoretic insight on intrinsic error-correction properties of gridcells and their related large capacity may indeed be central ingredients to produce grid cells.
In fact, there are a number of key properties of the grid cell code elucidated earlier by theoreticalarguments, including but not limited to path integration, that we hypothesize may form a sufficient andbiologically relevant set for the emergence of grid cells: 1) non-negative activations; 2) equivariantpopulation responses (i.e., the population response always lies on a hypersphere); 3) a path integrating(PI) code or in other words, translation invariant representations [17, 7]; 4) high representationalcapacity [6, 48, 34]; 5) intrinsic error-correcting capabilities [6, 48]; and finally, 6) uniformlydistributed (whitened) and low spatial information per cell, such that the total spatial information ofthe grid code should be equally distributed across all cells.
We suggest that several of these properties (e.g. excluding the more specific property 3)) are extremelygeneral features of neural codes, and incorporating them into ANN models of neural circuits couldincrease their ability to make de novo rather than post hoc predictions, and remove the need to usesynthetic and unrealistic choices with fine hyperparameter tuning (such as DOG and DOS readouts).
In sum, the findings here argue that the central contribution of scientific interest when building andstudying ANN models of the brain that reproduce specific tuning curves is not that the model producedthe curves (after all, given the expressive power of deep networks, it is no surprise that trainingthem to effectively generate a given tuning will in fact succeed), but rather to explore and carefullycharacterize conditions and constraints under which the particular tuning does and does not emerge,to consider whether the constraints align with biological constraints, and to consider what principlescan be extracted from them.
When not predicting specific tuning curves, but rather population-levelresponse dynamics, the results are expected to be more robust to specific implementational choicesand thus more durable.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] We do notfeel our paper has potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No] We haven’t yet had time to collectthis information, but we will add this information to the final version if accepted.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes](c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether
and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]