Paper:
Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural networks in sensory and cognitive systems. However, this interaction between fields is less developed in the study of motor control. In this work, we develop a virtual rodent as a platform for the grounded study of motor activity in artificial models of embodied control. We then use this platform to study motor activity across contexts by training a model to solve four complex tasks. Using methods familiar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using a neuroethological approach to characterize motor activity relative to the rodent's behavior and goals. We find that the model uses two classes of representations which respectively encode the task-specific behavioral strategies and task-invariant behavioral kinematics. These representations are reflected in the sequential activity and population dynamics of neural subpopulations. Overall, the virtual rodent facilitates grounded collaborations between deep reinforcement learning and motor neuroscience.

Review:
The authors use a virtual rodent to study flexible behavior, a key concept at the intersection of AI and neuroscience. “Flexibility” is a common buzzword in this field used to describe what it is that mammals do better than deep nets, even as deep nets can outperform them on any one task in isolation with sufficient training. As the authors note, previous work addresses “cognitive flexibility” (Yang et al. 2019) in isolation without motor commands, or studies motor dynamics of real mammals but without any task flexibility—occluding the actual advantages that mammals have over machines. However, this work goes the full mile in requiring the agent to implement flexible cognition using a complex body in a physical environment with egocentric stimuli. From a strictly AI point of view, I could imagine arguments that this is not sufficiently “novel,” in that a lot of deep RL researchers have already trained agents with complex joint structures to succeed in such virtual worlds. But the real substance of the paper is the application of data analysis techniques imported directly from neuroscience. It is important to suss out what features of neural dynamics are peculiarities of biological computation and which are natural consequences of the tasks being performed. I find the paper to be overall well motivated.
The analyses were well ordered, starting with a high-level characterization of behavior that showed a hierarchy of features, where “low-level” or “fast” features are mixed across tasks while “high-level” or “slow” features are more task specific. They then moved on to find “neural correlates” at the behavior and task levels, driving home their primary point that core layers encode high-level variables in a way not shared across tasks while policy layers implement reusable core motor functions that are used in multiple contexts. This observation coincides with the measurable frequency content of core and policy activity, which is demonstrably faster in policy layers. Finally, the authors show a visualization of the dynamical systems that underlie these computations, with videos that very clearly illustrate how core and policy dynamics dissociate to specialize in cognition vs. motor generation. This part is where they really leverage the artificial setting, where the exact dynamics are known and do not need to be inferred from noisy neural data.
While the end result of the analyses led to convincing visuals that made the intended point, the details of the analysis themselves were hard to follow and evaluate for rigor. An appendix explaining with notation how the behavioral “features” were computed, for example, would have been helpful. I acknowledge that it is a feature of the paper that so many different analysis techniques from neuroscience—dynamical systems to representation analysis to lesion studies—were applied, and the sheer breadth of the analysis was impressive, if difficult to evaluate.
Although I recognize that to do so would be difficult, I think what would add a lot to this paper would be side-by-side comparisons of real neural data with “neural” data from the artificial rodent. Perhaps the behavior would have to be modified so that some tasks match specific neuroscience paradigms, or the sample size for analysis limited to what is realistic for experimental data. It would be neat to see if, say, preparatory activity signals could emerge from training such an artificial rodent to do many tasks, including a head-fixed timed lever pull or some other classic paradigm. Maybe future work can draw a more exact analogy between policy/core layers and specific brain regions.
As to topic fit, this is niche in the context of ICLR; however, showing how state-of-the-art deep RL techniques can be applied to neuroscientific questions may be of interest to machine learning researchers even if they are not also neuroscientists.
Overall, I thoroughly enjoyed this paper and look forward to the new kinds of research that sprout out of it!