D EEP NEUROETHOLOGY OF A VIRTUAL RODENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artiﬁcial neural networks in sensory and cognitive systems.
However, this interaction between ﬁelds is less developed in the study of motor control.
In this work, we develop a virtual rodent as a platform for the grounded study of motor activity in artiﬁcial models of embodied control. We then use this platform to study how motor activity differs across contexts by training a model to solve four complex tasks.
Using methods familiar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using a neuroethological approach to characterize motor activity relative to the rodent’s behavior and goals.
We ﬁnd that the model uses two classes of behavioral representations which respectively encode the task-speciﬁc behavioral strategies and task-invariant behavioral kinematics. These representations are reﬂected in the sequential activity and population dynamics of neural subpopulations.
Overall, the virtual rodent facilitates grounded collaborations between deep reinforcement learning and motor neuroscience.
I NTRODUCTION
Animals have nervous systems that allow them to coordinate their movement and perform a diverse set of complex behaviors.
Mammals, in particular, are generalists in that they use the same general neural network to solve a wide variety of tasks.
This ﬂexibility in adapting behaviors towards many different goals far surpasses that of robots or artiﬁcial motor control systems.
Hence, studies of the neural underpinnings of ﬂexible behavior in mammals could yield important insights into the classes of algorithms capable of complex control across contexts and inspire algorithms for ﬂexible control in artiﬁcial systems.
Recent efforts at the interface of neuroscience and machine learning have sparked renewed interest in constructive approaches in which artiﬁcial models that solve tasks similar to those solved by animals serve as normative models of biological intelligence.
Researchers have attempted to leverage these models to gain insights into the functional transformations implemented by neurobiological circuits, prominently in vision (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014; Kar et al., 2019), but also increasingly in other areas, including audition (Kell et al., 2018) and navigation (Banino et al., 2018; Cueva & Wei, 2018).
Efforts to construct models of biological locomotion systems have informed our understanding of the mechanisms and evolutionary history of bodies and behavior (Grillner et al., 2007;
Ijspeert et al., 2007; Ramdya et al., 2017; Nyakatura et al., 2019).
Neural control approaches have also been applied to the study of reaching movements, though often in constrained behavioral paradigms (Lillicrap & Scott, 2013), where supervised training is possible (Sussillo et al., 2015; Michaels et al., 2019).
While these approaches model parts of the interactions between animals and their environments (Chiel & Beer, 1997), none attempt to capture the full complexity of embodied control, involving how an animal uses its senses, body and behaviors to solve challenges in a physical environment. The development of models of embodied control is valuable to the ﬁeld of motor neuroscience, which typically focuses on restricted behaviors in controlled experimental settings.
It is similarly valuable to AI research, in which ﬂexible models of embodied control could be applicable to robotics. Here, we introduce a virtual model of a rodent to facilitate grounded investigation of embodied motor systems capable of solving multiple cognitive challenges.
The virtual rodent affords a new opportunity to directly compare principles of artiﬁcial control to biological data from real-world rodents, which are more experimentally accessible than humans. We draw inspiration from emerging deep reinforcement learning algorithms which now allow artiﬁcial agents to perform complex and adaptive movement in physical environments with sensory information that is increasingly similar to that available to animals (Peng et al., 2016; 2017; Heess et al., 2017; Merel et al., 2019a;b).
Similarly, our virtual rodent exists in a physical world, equipped with a set of actuators that must be coordinated for it to behave effectively.
It also possesses a sensory system that allows it to use visual input from an egocentric camera located on its head and proprioceptive input to sense the conﬁguration of its body in space. There are several questions one could answer using the virtual rodent platform.
Here we focus on the problem of embodied control across multiple tasks. While some efforts have been made to analyze neural activity in reduced systems trained to solve multiple tasks (Song et al., 2017; Yang et al., 2019), those studies lacked the important element of motor control in a physical environment. Our rodent platform presents the opportunity to study how representations of movements as well as sequences of movements change as a function of goals and task contexts. To address these questions, we trained our virtual rodent to solve four complex tasks within a physical environment, all requiring the coordinated control of its body.
We then ask “Can a neuroscientist understand a virtual rodent?”
– a more grounded take on the originally satirical “Can a biologist ﬁx a radio?”
(Lazebnik, 2002) or the more recent “Could a neuroscientist understand a microprocessor?”
(Jonas & Kording, 2017).
We take a more sanguine view of the tremendous advances that have been made in computational neuroscience in the past decade, and posit that the supposed ‘failure’ of these approaches in synthetic systems is partly a misdirection.
Analysis approaches in neuroscience were developed with the explicit purpose of understanding sensation and action in real brains, and often implicitly rooted in the types of architectures and processing that are thought relevant in biological control systems.
With this philosophy, we use analysis approaches common in neuroscience to explore the types of representations and dynamics that the virtual rodent’s neural network employs to coordinate multiple complex movements in the service of solving motor and cognitive tasks.
A PPROACH 2. V IRTUAL RODENT BODY
We implemented a virtual rodent body (Figure 1) in MuJoCo (Todorov et al., 2012), based on measurements of laboratory rats (see Appendix A.1).
The rodent body has 38 controllable degrees of freedom.
The tail, spine, and neck consist of multiple segments with joints, but are controlled by tendons that co-activate multiple joints (spatial tendons in MuJoCo).
Upon publication, we will release the rodent model.
The virtual rodent is equipped with a set of proprioceptive inputs as well as “raw” egocentric RGBcamera (64 × 64 pixels) input from a head-mounted camera. The proprioceptive inputs include internal joint angles and angular velocities, the positions and velocities of the tendons that provide actuation, egocentric vectors from the root (pelvis) of the body to the positions of the head and paws, a vestibular-like upright orientation vector, touch or contact sensors in the paws, as well as egocentric acceleration, velocity, and 3D angular velocity of the root. 2. V IRTUAL RODENT TASKS
We implemented four tasks adapted from previous work in deep reinforcement learning and motor neuroscience (Merel et al., 2019a; Tassa et al., 2018; Kawai et al., 2015) to encourage diverse motor behaviors in the rodent.
The tasks are as follows: (1) Run along a corridor, over “gaps”, with a reward for traveling along the corridor at a target velocity (Figure 2A).
(2) Collect all the blue orbs in a maze, with a sparse reward for each orb collected (Figure 2B).
(3) Escape a bowl-shaped region by traversing hilly terrain, with a reward proportional to distance from the center of the bowl (Figure 2C).
(4) Approach orbs in an open ﬁeld, activate them by touching them with a forepaw and touch them a second time after a precise interval of 800ms with a tolerance of ± 100ms; there is a time-out period if the touch is not within the tolerated window and rewards are provided sparsely on the ﬁrst and second touch (Figure 2D).
We do not provide the agent with a cue or context indicating its task.
Rather, the agent must infer the task from the visual input and behave appropriately.
2. T RAINING A MULTI - TASK POLICY
the dashed arrow indicating no backpropogation along this path during training) along with shortcut paths from the proprioceptive observations as well as encoded features.
The policy module consists of one or more stacked LSTMs (with or without skip connections) which then produce the actions via a stochastic policy. Emboldened by recent results in which end-to-end RL produces a single terrain-adaptive policy (Peng et al., 2016; 2017; Heess et al., 2017), we decided to train a single model on the multiple motor-control-reliant tasks.
To train a single policy to perform all four tasks, we used an IMPALAstyle setup for actor-critic DeepRL (Espeholt et al., 2018); parallel workers collected rollouts, logged them to a replay, from which a central learner sampled data to perform updates.
The value-function critic was trained using off-policy correction via V-trace.
To update the actor, we used a variant of
MPO (Abdolmaleki et al., 2018) where the E-step is performed using advantages determined from the empirical returns and the value-function, instead of the Q-function (Anonymous, 2019).
Empirically, we found that the “escape” task was more challenging to learn during interleaved training relative to the other tasks.
Consequently, for the results we present, we trained a single-task expert on the escape task and trained the multi-task policies using kickstarting for that task (Schmitt et al., 2018), with a weak coefﬁcient (.001 or .005).
Kickstarting on this task made the seeds more reliably solve all four tasks, facilitating comparison of the multi-task policies with different architectures (i.e. the policy having 1, 2, or 3 layers, with or without skip connections across those layers).
The procedure yields a single neural network that is capable of using visual inputs to determine how to behave as well as coordinating its body to move in ways required to solve the tasks.
See video examples of a single policy solving episodes of each task: gaps, forage, escape, and two-tap.
A NALYSIS
We analyzed the virtual rodent’s neural network activity in conjunction with its behavior to attempt to characterize how it solves multiple tasks. We apply analyses and perturbation techniques adapted from neuroscience, where a range of techniques have been developed to highlight the properties of real neural networks. Biological neural networks have been hypothesized to control, select, and modulate movement through a variety of debated mechanisms, ranging from explicit neural representations of muscle forces and behavioral primitives, to more abstract production of the dynamics that could enable movement (Churchland et al., 2012; Graziano, 2006; Kalaska, 2009).
A challenge with nearly all of these models however is that they have largely been inspired by ﬁndings from individual behavioral tasks, making it unclear how to generalize them to a broader range of naturalistic behaviors. To provide insight into mechanisms underlying movement in the virtual rodent, and to potentially give insight by proxy into the mechanisms underlying behavior in real rats, we thus systematically tested how the different network layers encoded and generated different aspects of movement. For all analyses we logged the virtual rodent’s kinematics, joint angles, computed forces, sensory inputs, and the cell unit activity of the LSTMs in core and policy layers during 25 trials per task from each network architecture.
3. V IRTUAL RODENTS EXHIBIT BEHAVIORAL FLEXIBILITY .
We began our analysis by quantitatively describing the behavioral repertoire of the virtual rodent. A challenge in understanding the neural mechanisms underlying behavior is that it can be described at many timescales.
On short timescales, one could describe rodent locomotion using a set of actuators that produce joint-speciﬁc patterns of forces and kinematics.
However on longer timescales, these force patterns are organized into coordinated, re-used movements, such as running, jumping, and turning. These movements can be further combined to form behavioral strategies or goal-directed behaviors.
Relating neural representations to motor behaviors therefore requires analysis methods that span multiple timescales of behavioral description.
To systematically examine the classes of behaviors these networks learn to generate and how they are differentially deployed across tasks, we developed sets of behavioral features that describe the kinematics of organismal-scale behavior of the animal on fast (5-25 Hz), intermediate (1-25 Hz) or slow (0.3-5 Hz) timescales (Appendix
A.2 A.3 ).
As validation that these features provided a feature set capable of distinguishing between different behaviors, embedding these features using tSNE (Maaten & Hinton, 2008) produced a behavioral map in which virtual rodent behaviors, including rearing, jumping, running, climbing and spinning, were segregated to different regions of the map (Figure 4B)(see video).
Interestingly, we found that the many of the behaviors that the virtual rodent generated were shared across tasks, suggesting that the agent exhibited behavioral ﬂexibility. On short timescales, patterns of activation of both low level motor features like joint speed and actuator forces, and longer timescale behavioral features were largely shared across tasks (Figure 4C).
In contrast, longer timescale behavioral features were more differentiated across tasks, suggesting that the network learned to adapt similar movements in a selective manner for different tasks.
3. N ETWORKS PRIMARILY REFLECT BEHAVIORS , NOT FORCES
We next examined the neural activity patterns underlying the virtual rodent’s behavior, to test if networks produced behaviors through explicit representations of forces, kinematics or behaviors.
As expected, core and policy units operate on distinct timescales (See Appendix A.3, Figure 9).
Units in the core typically ﬂuctuated over timescales of 1-10 seconds, likely representing variables associated with context and reward.
In contrast, units in policy layers were more active over subsecond timescales, potentially encoding motor and behavioral features.
To quantify which aspects of behavior were encoded in the core and policy layers, and how these encoding patterns varied across layers, we turned to representational similarity analysis (RSA) (Kriegeskorte et al., 2008; Kriegeskorte & Diedrichsen, 2019). RSA provides a global measure of how well different features are encoded in different layers of a neural network by analyzing the geometries of network activity upon exposure to several stimuli, such as objects or odorants.
To apply RSA, ﬁrst a representational similarity (or equivalently, dissimilarity) matrix is computed that quantiﬁes the similarity of neural population responses to a set of stimuli. To test if different neural populations show similar stimulus encodings, these similarity matricies can then be directly compared across different network layers. Multiple metrics, such as the matrix correlation or dot product can be used to compare these neural representational similarity matricies.
Here we used the linear centered kernel alignment (CKA) index, which shows invariance to orthonormal rotations of population activity (Kornblith et al., 2019). RSA can also be used to directly test how well a particular stimulus feature is encoded in a population.
If each stimuli can be quantitively described by one or more feature vectors, a similarity matrix can also be computed across the set of stimuli themselves. The strength of encoding of a particular set of features can by measured by comparing the correlation of the stimulus feature similarity matrix and the neuronal similarity matrix. The correlation strength directly reﬂects the ability of a linear decoder trained on the neuronal population vector to distinguish different stimuli (Kriegeskorte & Diedrichsen, 2019).
Unlike previous applications of RSA in the analysis of discrete stimuli such as objects, (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014) behavior evolves continuously. To adapt RSA to behavioral analysis, we partitioned time by discretizing each behavioral feature into 50 clusters (Appendix A.4).
As expected, RSA revealed that core and policy layers encoded distinct behavioral features.
Policy layers contained greater information about fast timescale behaviors in a manner that was largely conserved across layers, while core layers showed more moderate behavioral encoding that was stronger for slow behavioral features (Figure 5B,C).
This difference in encoding was largely consistent across all architectures tested (Figure 5D). More interestingly, the feature encoding of policy networks was somewhat consistent with the emergence of a hierarchy of behavioral abstraction. In networks trained with three policy layers, representations were distributed in timescales across layers, with the last layer (Policy 2) showing stronger encoding of fast behavioral features, and the ﬁrst layer (Policy 0) instead showing stronger encoding of slow behavioral features.
However, policy layers, even those close to the motor periphery, did not show strong explicit encoding of behavioral kinematics or forces.
We then investigated the degree to which the rodent’s neural networks were able to use the same neural representations to produce behaviors, such as running or spinning, that were shared across tasks.
Embedding population activity into two-dimensions using multidimensional scaling revealed that core neuron representations were highly distinct across all tasks, while policy layers contained more overlap (Figure 6A), suggesting that some behavioral representations were re-used.
Comparison of representational similarity matricies for behaviors that were shared across tasks revealed that policy layers tended to possess a relatively similar encoding of behavioral features, especially fast behavioral features, over tasks (Figure 6C; Appendix A.4). This was validated by inspection of neural activity during individual behaviors shared across tasks Appendix A.5, Figure 10).
Core layer representations across almost all behavioral categories were more variable across tasks, consistent with encoding a longer timescale representation of behavioral sequences or abstract encodings of task variables.
Interestingly, when comparing this cross-task encoding similarity across architectures, we found that one layer networks showed a marked increase in the similarity of behavioral encoding across tasks (Figure 6D).
This suggests that in networks with lower computational capacity, animals must rely on a smaller, shared behavioral representation across tasks.
3. N EURAL POPULATION
[Figure] Representational similarity analysis revealed that on policy and core activity was structured into different subspaces for different behaviors. However RSA also revealed that these networks did not posess explicit low-level representations of forces or kinematics, suggesting networks used different mechanisms to directly generate behavior.
Inspection of the activity patterns underlying individual behaviors in the two-tap task showed peak activity in core and policy units was sequentially organized (Figure 7), uniformly tiling time between both taps of the two-tap sequence.
This sequential activation in policy networks was observed across tasks and behaviors, including during running (see video) where, consistent with policy networks encoding short-timescale kinematic features in a task-invariant manner, neural activity sequences were largely conserved across tasks (See Appendix A.5, Figure 10).
Interestingly, we also observed that neural variability is reduced during the inter- trial interval when more stereotyped behavior is demanded by the two-tap task, which may have bearing on classic results in motor control (See Appendix A.6, Figure 11).
To systematically examine the types of sequential or dynamical activity present in core and policy networks, we examined the dynamics of the networks’ population vector over time.
We ﬁrst applied principal components analysis (PCA) to the activity during the performance of single tasks, and visualized the gradient of the population vector as a vector ﬁeld.
Figure 8A shows such a vector ﬁeld representation of the ﬁrst two principal components of the core and ﬁnal policy layer during the two-tap task.
We generated vector ﬁelds by discretizing the PC space into a two-dimensional grid and calculating the average neural activity gradient with respect to time for each bin.
The vector ﬁelds showed strong signatures of rotational dynamics across all layers.
To quantify the frequency of these dynamical patterns, we turned to jPCA, a dimensionality reduction method that extracts rotational dynamics in neural activity by ﬁtting a rotational linear dynamical system to neural data (Churchland et al., 2012).
The resulting jPCs form an orthonormal basis that spans the same space as the ﬁrst six traditional PCs, while maximally emphasizing rotational dynamics.
Figure 8B shows the vector ﬁelds of the ﬁrst two jPC planes for the core and ﬁnal policy layers along with their characteristic frequency.
Consistent with our previous ﬁndings, jPC planes in the core have lower characteristic frequencies than those in policy layers across tasks (Figure 8C).
The jPC planes also individually explained a large percentage of total neural variability (Figure 8D).
These rotational dynamics in the policy and core jPC planes were respectively associated with the production of behaviors and the reward structure of the task.
For example, in the two-tap task, rotations in the fastest jPC plane in the core were concurrent with the approach to reward, while rotations in the second fastest jPC were concurrent with long timescale transitions between running to the orb and performing the two-tap sequence.
Similarly, the fastest jPC in policy layers was correlated with the phase of running, while the second fastest was correlated with the phase of the two-tap sequence (see video).
This trend of core and policy neural dynamics respectively reﬂecting behavioral and task-related features was also present in other tasks.
For example, in the maze forage task, the ﬁrst two jPC planes in the core respectively correlated with reaching the target orb and discovering the location of new orbs, while those in the policy were correlated with low-level locomotor features such as running phase (see video).
Along with our representational similarity analysis, these ﬁndings further support a model in which the core layer transforms sensory information into a sensorimotor contextual signal in a task-speciﬁc manner while the policy uses this signal and sensory inputs to generate appropriate behaviors in a more task-independent fashion.
For a more complete set of behaviors with neural dynamics visualizations overlain, see Appendix A.7.
3. N EURAL PERTURBATIONS CORROBORATE DISTINCT ROLES ACROSS LAYERS
To causally demonstrate the differing roles of core and policy units in respectively encoding taskrelevant features and movement, we performed perturbations of different neuronal subsets in the two-tap task.
We identiﬁed two stereotyped behaviors (rears and spinning jumps) that were reliably used in two different seeds of the agent to reach the orb in the task.
We ranked neurons according to the degree of modulation of their z-scored activity during the performance of these behaviors.
We then inactivated subsets of neurons by clamping their activity to their mean values between the ﬁrst and second taps and observed the effects of inactivation on trial success and behavior.
In both seeds analyzed, inactivation of policy units had a stronger effect on motor behavior than the inactivation of core units.
For instance, in the two-tap task, ablation of 64 neurons in the ﬁnal policy layer disrupts the performance of the spinning jump (Appendix A.8 Figure 12B video).
In contrast, ablation of behavior-modulated core units did not prevent the production of the behavior, but mildly affected the way in which the behavior is directed toward objects in the environment.
For example, ablation of a subset of core units during the performance of a spinning jump had a limited effect, but sometimes resulted in jumps that missed the target orbs (video; See Appendix A.8, Figure 12C).
We also performed a complementary perturbation aimed to elicit behaviors by overwriting the cell state of neurons in each layer with the time-varying average trajectory of neural activity measured during natural performance of a target behavior.
The efﬁcacy of stimulation was found to depend on the gross body posture and behavioral state of an animal, but was nevertheless successful in some cases.
For example, during the two-tap sequence, we were able to elicit spinning movements common to searching behaviors in the forage task (video; See Appendix A.8, Figure 12D, E).
The efﬁcacy of this activation was more reliable in layers closer to the motor output (Figure 12D).
In fact, activation of core units rarely elicited spins, but rather elicited sporadic dashes reminiscent of the searching strategy of many models during the forage task (video).
D ISCUSSION
For many computational neuroscientists and artiﬁcial intelligence researchers, an aim is to reverseengineer the nervous system at an appropriate level of abstraction.
In the motor system, such an effort requires that we build embodied models of animals equipped with artiﬁcial nervous systems capable of controlling their synthetic bodies across a range of behavior.
Here we introduced a virtual rodent capable of performing a variety of complex locomotor behaviors to solve multiple tasks using a single policy.
We then used this virtual nervous system to study principles of the neural control of movement across contexts and described several commonalities between the neural activity of artiﬁcial control and previous descriptions of biological control. A key advantage of this approach relative to experimental approaches in neuroscience is that we can fully observe sensory inputs, neural activity, and behavior, facilitating more comprehensive testing of theories related to how behavior can be generated. Furthermore, we have complete knowledge of the connectivity, sources of variance, and training objectives of each component of the model, providing a rare ground truth to test the validity of our neural analyses.
With these advantages in mind, we evaluated our analyses based on their capacity to both describe the algorithms and representations employed by the virtual rodent and recapitulate the known functional objectives underlying its creation without prior knowledge. To this end, our description of core and policy as respectively representing value and motor production is consistent with the model’s actor-critic training objectives. But beyond validation, our analyses provide several insights into how these objectives are reached.
RSA revealed that the cell activity of core and policy layers had greater similarity with behavioral and postural features than with short-timescale actuators. This suggests that the representation of behavior is useful in the moment-to-moment production of motor actions in artiﬁcial control, a model that has been previously proposed in biological action selection and motor control (Mink, 1996; Graziano, 2006). These behavioral representations were more consistent across tasks in the policy than in the core, suggesting that task context and value activity in the core engaged task-speciﬁc behavioral strategies through the reuse of shared motor activity in the policy.
Our analysis of neural dynamics suggests that these reused motor activity patterns are often organized as sequences. Speciﬁcally, the activity of policy units uniformly tiles time in the production of several stereotyped behaviors like running, jumping, spinning, and the two-tap sequence.
This ﬁnding is consistent with reports linking sequential neural activity to the production of stereotyped motor and task-oriented behavior in rodents (Berke et al., 2009; Rueda-Orozco & Robbe, 2015), including during task delay periods (Akhlaghpour et al., 2016), as well as in singing birds (Albert &
Margoliash, 1996; Hahnloser et al., 2002). Similarly, by relating rotational dynamics to the virtual rodent’s behavior, we found that different behaviors were seemingly associated with distinct rotations in neural activity space that evolved at different timescales. These ﬁndings are consistent with a hierarchical control scheme in which the sequences and dynamics in the policy that mediate the generation of reused behaviors are modulated and patterned by sensorimotor signals from the core.
This work represents an early step toward the constructive modeling of embodied control for the purpose of understanding the neural mechanisms behind the generation of behavior.
Incrementally and judiciously increasing the realism of the model’s embodiment, behavioral repertoire, and neural architecture is a natural path for future research.
Our virtual rodent possesses far fewer actuators and touch sensors than a real rodent, uses a vastly different sense of vision, and lacks integration with olfactory, auditory, and whisker-based sensation (see Zhuang et al., 2017).
While the virtual rodent is capable of locomotor behaviors, an increased diversity of tasks involving decision making, memory-based navigation, and working memory could give insight into “cognitive” behaviors of which rodents are capable.
Furthermore, biologically-inspired design of neural architectures and training procedures should facilitate comparisons to real neural recordings and manipulations.
We expect that this comparison will help isolate residual elements of animal behavior generation that are poorly captured by current models of motor control, and encourage the development of artiﬁcial neural architectures that can produce increasingly realistic behavior.
A UTHOR C ONTRIBUTIONS
Contributions masked for anonymity.
A CKNOWLEDGMENTS
The rodent skeleton that we used as a reference for designing the virtual rodent body was purchased from leo3Dmodels on TurboSquid.
Other acknowledgements masked for anonymity.
A A PPENDIX A. R AT MEASUREMENTS
In order to construct the virtual rodent model, we obtained the mass and lengths of the largest body segments that inﬂuence the physical properties of the virtual rodent.
First, we dissected cadavers of two female Long-Evans rats, and measured the mass of relevant limb segments and organs.
Next, we measured the lengths of body segments over the skin of animals anesthetized with 2% v/v isoﬂurane anesthesia in oxygen.
We conﬁrmed that these skin based measurements approximated bone lengths by measuring bone lengths in a third cadaver.
The care and experimental manipulation of all animals were reviewed and approved by the appropriate Institutional Animal Care and Use Committee.
[Table]
Skull
[Table] [Table]
[Table]
[Table]
Scapula L 3.
Humerus L [Table]
[Table]
/ulna L 2. Forepaw L
[Table]
Scapula R 2.
Humerus R 6.
Radius/ulna R 2.
[Table] [Table] 17.7 ± 1. Wrist to ﬁnger L 15.
16.6 ± 0.
Wrist to pad L 6.
6.4 ± 1. Wrist to olecranon L 29. 31.9 ± 1. Humerus L 31.
29.2 ± 2.
Scapula L 22.
26.2 ± 2.
Wrist to claw R 16. 17.0 ± 0. Wrist to ﬁnger R 14. 15.1 ± 1.
Wrist to pad R 5.
5.7 ± 0. Wrist to olecranon R 30. 31.2 ± 1. Humerus R 28.
28.1 ± 3.
Scapula R 23.
26.4 ± 2.
Headcap width Headcap length
Skull width 38.
25.1 ± 6.
Skull length 51.
[Table]
We generated behavioral features describing whole-body pose and kinematics of the virtual rodent on fast, intermediate, and slow temporal scales.
To describe the whole-body pose, we took the top 15 principal components of the virtual rodent’s joint angles and joint positions to yield two 15 dimensional sets of eigenpostures (Stephens et al., 2008). We combined these into a 30 dimensional set of postural features. To describe the animal’s whole-body kinematics, we computed the continuous wavelet transform of each eigenposture using a Morlet wavelet spanning 25 frequencies. This yielded a 375 dimensional time-frequency representation of the underlying kinematics.
We then computed the top 15 principal components of each 375 dimensional time-frequency representation. This yielded a 30 dimensional representational description of the animal’s behavioral dynamics. To facilitate comparison of organismal behavior to neural representations on different timescales, we generated three sets of Morlet wavelet frequencies on 1 to 25 Hz (intermediate), 0.3 to 5 Hz (slow) or 5-25 Hz (fast) timescales. In separate work, we have found that combining postural and dynamical information improves separation of animal behaviors in behavioral embeddings.
Therefore, we combined postural and dynamical features, the later on intermediate timescales, to yield a 60 dimensional set of behavioral features that we used to map the animal’s behavior using tSNE (Figure 4C) (Berman et al., 2014). tSNEs were made using the Barnes-Hut approximation with a perplexity of 30. A. P OWER SPECTRAL DENSITY OF CORE AND POLICY LAYERS method using a 10 s window size and 5 s overlap.
We used representational similarity analysis to compare the population representations across network layers and to different subsets of behavioral features. Representational similarity analysis has in the past been used to compare neural population responses in tasks where behavioral stimuli are discrete, for instance corpuses of objects or faces (Kriegeskorte et al., 2008; Kriegeskorte & Diedrichsen, 2019). A challenge in scaling such approaches to neural analysis in the context of behavior is that behavior unfolds continuously in time.
It is thus a priori unclear how to discretize behavior into discrete chunks in which to compare representations. Formally, we deﬁned a set of eight different behavioral feature sets B i =1 ... 8 describing different aspects of the behavior of the animal on different timescales. These included instantaneous features such as joint angles, the angular speed of the joint angles, the eigenpostures of the animal, and the actuator forces, behavioral kinematics consisting of the wavelet transform of the eigenpostures computed on slow, fast, and intermediate timescales, and behavioral features, which consisted of both behavioral kinematics and the eigenposture features. Each feature set is a matrix B i ∈
R Mxq i where M is the number of timepoints in the experiment and q i is the number of features in the set. We discretized each set B i using k-means clustering with k = 50 to yield a partition of the timepoints in the experiment P i . Using the discretization deﬁned in P i , we can perform representational simirity analysis to compare the structure of population responses across neural network layers L m and L n or between a given network layer and features of the behavior B i . Following notation in (Kornblith et al., 2019) we let
X ∈ R kxp be a matrix of population responses across p neurons and the k behavioral categories in p B i . We let Y ∈ R nxq be either the matrix of population responses from q neurons in a distinct network layer, or a set of q features describing the behavior of the animal in the feature set B j . It is not necessary that the behavioral feature being compared, j , is the same as used to create the discretization partition, i , but we do so in all cases here.
After computing the response matricies in a given partition, we compared the representational structure of the matricies XX T and Y Y T . To do so, we compute the similarity between these matricies using the linear Centered Kernel Alignment index, which invariance under orthonormal rotations of the population activity. Following (Kornblith et al., 2019), for the CKA coefﬁent is: [Equation]
Where (cid:107) · (cid:107) F is the Frobenius norm.
For centered X and Y , the numerator is equivalent to the dot-product between the vectorized responses (cid:107) XY T (cid:107) F = (cid:104) vec( XX T ) , vec( Y Y T ) (cid:105) . For a given network layer L m , and a behavioral partition P i , we can denote XX T = D L m P i = D mi .
Similarly, for a given feature set B i , let D B i P i = D ii . Thus we are interested in characterizing both [Equation]
and
[Equation]
An additional challenge comes when restricting this analysis to comparing the neural representations of behavioral across different tasks T a , T b , where not all behaviors are necessarily represented in each task. To do so, we denote B i ( T a ) to be the set of behavioral clusters observed in task T a , and
B T a T b i = B i ( T a ) ∩ B i ( T a ) to be the set of behaviors used in each of the two tasks. We can then deﬁne a restricted partition of timepoints for each task P T a ,T b i or P T b ,T a i that includes only these behaviors, and compute the representational similarity between the same layer across tasks:
[Equation]
We have presented a means of performing representational similarity analysis across continuous time domains, where the natural units of discretization are unclear and likely multi-fold. While we focused on analyzing responses on the population level, it is likely that different subspaces of the population may encode information about distinct behavioral features at different timescales, which is still an emerging domain in representational similarity analysis. [Figure]
A. S TEREOTYPED BEHAVIOR INITIATION AND NEURAL VARIABILITY
During the execution of stereotyped behaviors, neural variability is reduced (Figure 11).
Recall that in our setting, neurons have no intrinsic noise, but inherit motor noise through observations of the state (i.e. via sensory reafference).
This effect loosely resembles, and perhaps informs one line of interpretation of the widely reported phenomenon of neural variability reducing with stimulus or task onset (Churchland et al., 2010).
Our reproduction of this effect, which simply emerges from training, suggests that variance modulation may partly arise from moments in a task that beneﬁt from increased behavioral precision (Renart & Machens, 2014).
A. N EURAL DYNAMICS VISUALIZED DURING TASK BEHAVIOR
For completeness, we provide links to videos of a few variants of neural dynamics for each task.
[Table] [Figure]